Answers That Survive an LLM Outage
When the LLM router died, a meta fallback event arrived and the visitor still got an answer. Here's the three-part chain behind it.
The LLM router died mid-test. Stream still connected, cursor blinking, but nothing arriving. I refreshed expecting an error. Instead the search box finished with a coherent answer about the blog's caching strategy. Turns out a visitor just asked the same question two minutes prior. The system replayed that response and kept the stream alive long enough to hide the failure.
That was the moment I stopped treating fallback as edge case handling. It was now the system working as designed.
event: meta
data: {"fallback":"cache", "provider":"..."}The /articles/ask endpoint was a straight pipe. Receive query, retrieve context, stream LLM response via SSE. Router down or model overloaded meant the client stared at an empty text area indefinitely. No timeout, no retry, no safety net. I'd seen it happen twice in production. Both times I manually restarted the service. Both times I told myself I'd add proper handling later. Later came after a week of intermittent failures during a model provider migration.
Three changes, one commit
Commit 151147a touched the entire /articles/ask endpoint. First, a mode parameter: search pulls from articles and knowledge, about answers as the site owner's persona from curated knowledge plus only strongly relevant articles, and mixed blends both. The about mode skips the full article index. It only returns context for posts that strongly match the query, keeping responses focused on the site owner's actual expertise instead of retrieving every tangentially related post.
Second, the answer cache moved to Redis. Previous cache lived in-process. Any restart meant cold starts for every query. Now it's Redis with AOF persistence and a 24-hour TTL. But the key scheme is what matters.
# bump the version when knowledge changes -> old keys become unreachable
redis-cli INCR askver:kn
# cache keys are built with the current version:
# askcache:kn:42:<query> <- 42 = value of askver:kn when the answer was made
# after a bump to 43, every ...:42:... key is never read again,
# then it destroys itself once the 24h TTL elapsesWhen a knowledge entry or article gets updated, the relevant namespace version counter increments via INCR. Old cache keys become unreachable because the version embedded in the key no longer matches current values. No SCAN, no mass DEL. Redis handles the rest: EXPIRE [4] source 4 destroys unreachable keys when their TTL elapses. The bump is atomic and takes O(1) time [5] source 5.
Fallback chain: from crash to continuity
The third change was the fallback chain. This is the part that saved the stream when the LLM died.
async def stream_answer(query, mode):
# 1. try the LLM
try:
async for chunk in llm_stream(query, mode):
yield chunk
except LLMError:
# 2. replay from cache
cached = get_cached_answer(query)
if cached:
yield sse_event("meta", {"fallback": "cache"})
yield cached
return
# 3. LLM-free persona from knowledge
answer = persona_answer(query, knowledge)
yield sse_event("meta", {"fallback": "no_llm"})
yield answerThe key design decision: cache hits became the failure mode, not the success path. Before this change, a cached hit meant the system skipped the LLM call. Now a cached hit means the LLM failed and the system is recovering. The distinction matters because it determines what gets logged, what triggers alerts, and what gets served to visitors.
Empty streams no longer count as success. If the LLM connects but sends zero tokens before dropping, that's a failure with no cache to replay from. The system falls through to the persona fallback.
This all works because Redis handles two jobs well: atomic counters for versioning [5] source 5 and automatic key expiration via TTL [4] source 4. SSE [6] source 6 keeps the connection alive through the fallback transition. The visitor sees a brief delay, then a complete answer. They never see an error. I could have done this with a time-based cache key and skipped the version counters entirely. It would have worked for 90% of cases. But the versioned approach means knowledge updates invalidate stale responses immediately, not after some arbitrary TTL window. That was worth the extra INCR.