The Marker That Hid Between Two Chunks

I was replaying a cached answer from the "Ask Adityo" chat in Redis, checking something unrelated, when I saw it: CTA_FLAG: YES sitting right there in the middle of a stored answer. Plain text, bold markers and all, served back to every visitor who hit that cached replay. The stripping code was running. It had been running since the previous commit. And yet the marker sailed straight through into the cache.

The feature is simple. The model writes an answer, and somewhere at the end it drops a CTA_FLAG: YES marker that my backend uses to decide whether to attach a call-to-action block. The marker is never meant to reach the user, so the streaming use case in use_cases.py strips it from the answer parts before yielding them. Or that was the theory. This was the code:

if any(_ASK_CTA_RE.search(d) for d in answer_parts):
    stripped = [_ASK_CTA_RE.sub(' ', d).strip() for d in answer_parts]
    yield AskEvent('cta_stripped', {'had_cta': has_cta_source})
    answer_parts = stripped

My first guess was wrong

I assumed the regex itself was broken. Case sensitivity, maybe, or the optional bold asterisks tripping it up. So I copied _ASK_CTA_RE into a scratch file, fed it a sample answer, and it matched perfectly. Fine. The regex works on the full answer.

Then I printed the actual deltas from a live stream and the problem fell into my lap. The model had split the marker across two chunks: one delta ended with CTA, the next one opened with _FLAG: YES. No single delta contains the full marker. And re.sub replaces occurrences of a pattern within one string; it cannot see a pattern that spans two different strings [2]. My per-delta loop was executing a correct regex against fragments, and a correct regex against a fragment matches nothing. The marker wasn't dodging the strip. The strip never had a chance to see it whole. It passed through the join intact, got stored, and came back on every replay.

This is the part that reframed the whole bug for me. I had been quietly assuming that one network chunk equals one logical piece of the answer. That assumption is wrong at the transport level. TCP hands you bytes, not messages, so a chunk can carry three complete events and half of a fourth; the frame delimiter is a blank line, not a chunk boundary [5]. Server-sent events are separated by a pair of newlines, and anything without an explicit event field arrives at the client as a plain message event [1]. The server can push data at any time [4], which means the model's token boundaries, the SSE framing, and the chunking your HTTP stack happens to do are three independent things. My marker spans a token boundary. Nothing promises that boundary lines up with anything else.

The two-layer fix

Deltas that contain a partial marker contribute no other readable text anyway, so stripping per-delta was doing nothing useful even when it matched. The new code only tidies up the matching deltas:

if any(_ASK_CTA_RE.search(d) for d in answer_parts):
    answer_parts = [_ASK_CTA_RE.sub(' ', d) for d in answer_parts]

That alone doesn't cover the split case. Crucially, the joined answer goes through the CTA extraction helper in the routes module after the stream ends, before the explanation-marker split and before the cache store. That joined-tail pass is the one that actually kills the marker, because by then the pattern exists in a single string. I also dropped the cta_stripped event, since announcing a cleanup nobody should ever notice felt like noise. Verified live: the joined stream and every individual delta come out marker-free now.

Where I land on this

The honest opinion: hand-stripping text markers out of LLM output is a hack, and the canonical way to get structured data from a model is JSON Schema structured outputs rather than parsing prose for magic strings [3]. If this feature grows a second flag, I'll switch. But for one boolean at the end of a streamed chat answer, I'd rather keep the marker and enforce one rule: sanitize at the last point before persistence, not at the point of arrival. Streaming sanitizers see fragments by nature. The cache sees the whole string. Strip there and you stop caring where the chunks fall.

The bug also left me with a general suspicion I didn't have before. Any regex run per-chunk is blind, in principle, to every pattern that could span two chunks. Not just markers. Entity names, code fences, URLs, anything the model might split mid-token. If you find yourself matching against stream fragments, assume the pattern will be cut in half eventually, then add a pass over the joined result as the safety net. In my case that net also had to exist before the Redis write, because a leaked marker in a cache doesn't flash once and disappear. It replays. Every time.

Sources

Sources consulted on September 1, 2026: Using server-sent events on MDN; re, regular expression operations in the Python docs; Structured Outputs from OpenAI; Server-sent events on MDN; and Streaming Responses: SSE, Chunks and Backpressure on multigrid.ai.

Related reading: One SSE Stream, Two AI Answers and That Time a Control Flag Showed Up in the Chat Bubble.