Pattern Resilience Advanced
Streaming Response
Your app reads the model’s reply as a stream and adds each chunk to the UI as it arrives, so users can watch the reply being written instead of looking at a spinner for ten seconds.
The problem
A user asks your assistant a question and then waits. The model is generating a 400-word reply, but your code is calling await response.json(), so nothing appears on screen until the very last token has been produced. The user looks at a blank space for ten seconds, and then a wall of text appears all at once.
The pattern
Instead of waiting for the full reply, your code can request a streaming response and read the body as it arrives. Most model APIs return either a raw text stream or Server-Sent Events (SSE), where each chunk contains a small piece of new text from the reply.
Problem
This is one of the most common mistakes I see in AI interfaces, and it’s avoidable. The model streamed that reply to your server token by token, and the API can stream it to the browser the same way. If your code waits for the whole reply before rendering any of it, the user can’t see that the app is working, and seeing the text arrive is what makes a slow response bearable.
A blank screen can make the app feel broken even when the reply is quick, because the user can’t tell that anything is happening. A streamed reply can take just as long to finish and still feel fast, because words keep appearing on screen the whole time.
Solution
The code works the same way in every framework. You get a reader from response.body, decode each chunk to text, pull the new tokens out of it, and append them to the message you’re rendering. Because the view is bound to the string you’re changing, the UI re-renders with each chunk, without any extra code.
Decoding is easy to get wrong. A chunk can end in the middle of a character, so you decode with a TextDecoder in streaming mode, { stream: true }, and then a character whose bytes are split across two chunks doesn’t come out garbled. Cancellation is easy to forget. If you pass an AbortController signal to the fetch, the user can cancel the stream, and if your server also stops the model call when the connection closes, a half-finished reply that the user has moved on from stops using up tokens. Once the first chunk arrives, the text replaces the Thinking Indicator you showed while there was nothing to render yet.
Example
The first example reads the stream in each framework, and the second parses SSE. Teams often get the markdown rendering and the scrolling wrong, so those have their own examples further down.
Reading the Stream
The loop reads each chunk from the response body, decodes it, and appends the text.
function useStreamingReply() {
const [text, setText] = useState('');
const [streaming, setStreaming] = useState(false);
const send = async (prompt, signal) => {
setText('');
setStreaming(true);
try {
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ prompt }),
signal,
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { value, done } = await reader.read();
if (done) break;
// stream: true keeps multi-byte characters intact across chunks
const chunk = decoder.decode(value, { stream: true });
setText(prev => prev + chunk);
}
} finally {
setStreaming(false);
}
};
return { text, streaming, send };
} <script setup>
import { ref } from 'vue';
const text = ref('');
const streaming = ref(false);
async function send(prompt, signal) {
text.value = '';
streaming.value = true;
try {
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ prompt }),
signal,
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { value, done } = await reader.read();
if (done) break;
text.value += decoder.decode(value, { stream: true });
}
} finally {
streaming.value = false;
}
}
defineExpose({ text, streaming, send });
</script>
<template>
<div class="reply">{{ text }}</div>
</template> <script>
let text = $state('');
let streaming = $state(false);
export async function send(prompt, signal) {
text = '';
streaming = true;
try {
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ prompt }),
signal,
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { value, done } = await reader.read();
if (done) break;
text += decoder.decode(value, { stream: true });
}
} finally {
streaming = false;
}
}
</script>
<div class="reply">{text}</div> class StreamingReply extends HTMLElement {
async send(prompt, signal) {
this.textContent = '';
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ prompt }),
signal,
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { value, done } = await reader.read();
if (done) break;
this.textContent += decoder.decode(value, { stream: true });
}
}
}
customElements.define('streaming-reply', StreamingReply); Parsing Server-Sent Events
Most model APIs send SSE frames instead of raw text, and each data: line in a frame contains a JSON delta with the new text. The parser below handles OpenAI’s Chat Completions format, which ends the stream with a data: [DONE] frame. Other APIs use different event formats, such as Anthropic’s typed events like content_block_delta, so the line that pulls out the token is different for each provider. A frame can be split across two chunks, so the parser has to buffer partial frames until the rest arrives.
async function* parseSSE(response) {
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
// Frames are separated by a blank line; keep the trailing partial in the buffer
const frames = buffer.split('\n\n');
buffer = frames.pop();
for (const frame of frames) {
const line = frame.split('\n').find(l => l.startsWith('data:'));
if (!line) continue;
const data = line.slice(5).trim();
if (data === '[DONE]') return;
const token = JSON.parse(data).choices?.[0]?.delta?.content;
if (token) yield token;
}
}
}
// Usage: for await (const token of parseSSE(response)) append(token);
// Aborting the fetch's signal makes reader.read() reject, which ends the loop.
Rendering Streaming Markdown
Models usually reply in markdown, and if you re-parse it on every token, broken tables and unclosed code fences will flash on screen. The example below handles code fences. It renders the text up to an unclosed fence as markdown, and shows the rest as plain text until the closing fence arrives. Tables need the same kind of check.
function StreamingMarkdown({ text, streaming }) {
const { safe, pending } = useMemo(() => {
if (!streaming) return { safe: text, pending: '' };
// If we're inside an unclosed code fence, render up to it as markdown
// and show the rest as plain text until the closing ``` arrives.
const fenceCount = (text.match(/```/g) || []).length;
if (fenceCount % 2 === 1) {
const lastFence = text.lastIndexOf('```');
return { safe: text.slice(0, lastFence), pending: text.slice(lastFence) };
}
return { safe: text, pending: '' };
}, [text, streaming]);
return (
<div className="markdown">
<Markdown>{safe}</Markdown>
{pending && <pre className="pending">{pending}</pre>}
</div>
);
}
Staying Anchored to the Latest Text
A long reply soon grows past the bottom of the visible area. The newest tokens should stay in view, but only while the user hasn’t scrolled up to read something earlier.
function useStickToBottom(ref, dep) {
const pinned = useRef(true);
useEffect(() => {
const el = ref.current;
const onScroll = () => {
const distance = el.scrollHeight - el.scrollTop - el.clientHeight;
pinned.current = distance < 40; // near the bottom = follow along
};
el.addEventListener('scroll', onScroll);
return () => el.removeEventListener('scroll', onScroll);
}, [ref]);
useEffect(() => {
if (pinned.current) ref.current.scrollTop = ref.current.scrollHeight;
}, [dep, ref]); // dep = the streaming text, so this runs on every chunk
}
Benefits
- The interface feels responsive even on a slow model, because text starts appearing as soon as the first tokens arrive and keeps coming.
- The delay feels much shorter, because the user spends that time reading the reply instead of looking at an empty screen.
- Users can start acting on the beginning of a long reply before the end has finished generating.
- Users are less likely to give up and leave, because they can see text arriving the whole time, so the app doesn’t look as if it’s locked up.
- It’s easy to add a Stop Generation control, because you already have the reader and the abort signal.
Tradeoffs
- Chunk boundaries don’t line up with characters or markdown syntax, so if you decode each chunk separately or render it right away as markdown, you’ll see garbled characters or broken markdown on screen.
- Streaming markdown is fiddly, because half-finished code fences and tables need special handling so they don’t flicker.
- Error handling gets harder, because a request can fail after the user has already read half a reply, so you need to keep the partial text instead of throwing it away.
- Auto-scrolling gets in the user’s way if your code doesn’t detect when they’ve scrolled up on purpose to read something.
- You can’t validate or post-process the complete response before showing it, because each token is already on screen as soon as it arrives.
- Some of the infrastructure between your server and the browser, such as certain proxies and compression layers, can buffer the chunks until the response closes, and then the whole reply arrives in the browser at once.
Summary
With a streaming response, your app reads the model’s reply as it’s produced and appends each chunk to the UI, so the user can read the reply as it’s written instead of waiting for the whole thing. That involves decoding chunks with a streaming TextDecoder, buffering partial SSE frames as you parse them, rendering only the finished part of the markdown, and scrolling to the bottom without taking over the user’s scroll position.
See also
- Thinking Indicator — Resilience
- Stop Generation — Resilience
- Message Thread — Components
- Loading State — Resilience
- Streaming API Responses - OpenAI
- Streaming - Anthropic
- Streams API - MDN
- Using Server-Sent Events - MDN