One SSE Stream, Two AI Answers
A chatbot needs both a quick summary and a full explanation from one LLM generation. The answer: marker-based splitting in the SSE stream, with separate cache storage and mid-stream parsing.
The "Ask Adityo" chatbot streams answers via SSE. User asks about the site, an answer appears token by token. But the UI needs two versions of the same answer: a short summary in the chat bubble, and a full explanation in an expandable panel. Both come from one LLM generation.
First attempt was to grab the first two sentences of the answer as the summary. Simple function, just slice the text. Problem: the LLM is inconsistent. Sometimes the summary is in the middle, sometimes at the end, sometimes nowhere at all. Slicing by sentence position broke in production because the model output changes every time.
So I forced it from the prompt.
Marker-based splitting
The system prompt now asks for two parts: the first line starts with RINGKAS, followed by a summary of at most two sentences (40 words); the next line starts with PENJELASAN, followed by a full explanation (220 words). Both separated by text markers. The LLM still generates one stream, but there's a text anchor that can be parsed.
Why not use structured output like JSON? Because this is SSE streaming. With JSON, you'd have to wait for the entire output to finish before you can parse it. With markers, parsing happens mid-stream. The moment PENJELASAN appears in a delta event, the frontend knows exactly where the split is.
On the backend, a new constant _ASK_ENHANCED_MARKER and a new event type AskEvent("enhanced", ...) carry the text after the PENJELASAN marker. The routes layer collects enhanced_parts separately from answer_parts, then the replay function emits the enhanced event to the client.
The cache payload grew from three fields (sources, answer, followups) to four, adding enhanced. The bubble stores only the summary without the marker. The enhanced text lives separately. Cache replays emit both events without ever leaking the raw marker to the UI.
Frontend handles the split in the delta case: when the PENJELASAN marker appears mid-stream, it gets stripped so it never shows in the bubble. A new enhanced case accumulates the text after the marker for the right panel. Backward compatible: old answers without the enhanced field fall back to plain bubble text.
Typed events versus markers
This sent me back to the SSE spec. The WHATWG standard already supports typed events via the event field. Authors can separate events by using different event types [1]. OpenAI uses semantic events like response.created and output_text.delta in their Responses API [2]. Vercel AI SDK has typed parts ranging from text-start to reasoning and custom [3]. NeuralSummary went with six separate typed events for their chat system [4].
But my situation is different. This isn't about different data types in one stream. It's about splitting one output into two views of the same payload. Typed events work well for separating text from reasoning from citations. For the case of one answer, two representations, marker-based splitting is lighter: no protocol-level changes, just coordination between prompt, parser, and renderer.
An IETF draft proposes a standard wire format for LLM inference streaming [5], but it's still a draft. And the standard focuses on raw inference tokens, not application-level splitting like what I needed.
Three layers must sync
One stream, two answer versions. The backend must detect when the marker appears and yield the enhanced event separately. The cache must store the split: summary in answer, detail in enhanced. The frontend must strip the marker mid-stream and render both panels. All three must stay in sync. If one misses, the user sees PENJELASAN text in the bubble, or the right panel stays empty.
One hundred ten test cases cover the new behavior: about-mode splitting, marker-free cache payload, single-frame replay from cache. A regression in any layer gets caught immediately.
Once the sync is in place, this scales to other structured output formats without changing the protocol. Add a new marker, yield a new event type, have the frontend handle it. The SSE transport stays the same old text/event-stream.
Sources:
[1] Server-Sent Events — WHATWG Living Standard
[2] OpenAI Responses API — Streaming
[3] Vercel AI SDK — Data Stream Protocol