Streaming Replies

Streaming shows the reply word by word as the model writes it, so users are not left staring at a blank screen.

Why Stream?

A long AI reply can take several seconds. Without streaming, the user waits and sees nothing. With streaming, the first words appear almost at once, which feels much faster, even though the total time is the same.

How Streaming Works

Your server adds stream: true to the request.
The AI API sends the reply in many small pieces, as server-sent events (SSE).
Each piece is a line starting with data: followed by JSON. The new text is in choices[0].delta.content.
The last line is data: [DONE].
Your server forwards each text piece to the browser, which adds it to the chat bubble.

Parsing the Stream Format

Here is what the raw stream looks like, and how to pull the text out of it. You can run this example as it is:

Example

javascript

const raw = [
  'data: {"choices":[{"delta":{"content":"Our"}}]}',
  'data: {"choices":[{"delta":{"content":" classes"}}]}',
  'data: {"choices":[{"delta":{"content":" start at 5 pm."}}]}',
  "data: [DONE]",
].join("\n\n");

function parseChunk(text) {
  const pieces = [];
  for (const line of text.split("\n")) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6);
    if (payload === "[DONE]") break;
    const delta = JSON.parse(payload).choices[0].delta.content;
    if (delta) pieces.push(delta);
  }
  return pieces;
}

const pieces = parseChunk(raw);
console.log(pieces);
console.log(pieces.join(""));

Output

[ 'Our', ' classes', ' start at 5 pm.' ]
Our classes start at 5 pm.

The Server Side

The network sends data in chunks that may cut a line in half, so the code keeps a buffer and only processes complete lines:

Example

javascript

// server.js (Express), key stays in environment variables
app.post("/api/chat-stream", async (req, res) => {
  const messages = [{ role: "system", content: SYSTEM }, ...cleanHistory(req.body.messages)];

  const ai = await fetch(`${process.env.AI_BASE_URL}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.AI_API_KEY}`,
    },
    body: JSON.stringify({ model: process.env.AI_MODEL, messages, stream: true }),
  });
  if (!ai.ok) return res.status(500).send("Sorry, please try again.");

  res.setHeader("Content-Type", "text/plain; charset=utf-8");
  const reader = ai.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop();                // keep the unfinished line for later
    for (const line of lines) {
      if (!line.startsWith("data: ")) continue;
      const payload = line.slice(6).trim();
      if (payload === "[DONE]") continue;
      const text = JSON.parse(payload).choices[0]?.delta?.content;
      if (text) res.write(text);         // send this piece to the browser
    }
  }
  res.end();
});

cleanHistory is the same filter from the previous lesson: it keeps only user and assistant messages and the last 10 of them.

The Browser Side

The browser reads the plain-text stream piece by piece and adds it to the bubble:

Example

javascript

async function sendMessage(messages, bubble) {
  const response = await fetch("/api/chat-stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  bubble.textContent = "";
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    bubble.textContent += decoder.decode(value, { stream: true });
  }
  return bubble.textContent;   // save this as the assistant message in history
}

Using textContent (not innerHTML) is important: it shows the AI's text as plain text, so any HTML in the reply cannot run in your page.

Tips

Tip	Why
Show a "typing..." indicator until the first piece arrives	Users know the bot is working
Disable the send button while streaming	Stops double messages
Save the full reply to history when the stream ends	The next request needs it
Handle a broken connection	Show "Connection lost, please retry"

Some hosting platforms buffer responses by default, which hides streaming. If replies arrive all at once after deploying, check your host's documentation on streaming responses.

Exercise

In a streamed chat completions response, what marks the end of the stream?

An empty JSON objectA line "data: [DONE]"The word "END" in the contentThe browser closing the tab