The follow-up question my AI chat couldn't answer

Amnesia on every turn

Someone asked the blog's AI chat a second question in a row. First: "What's your stance on Tailwind?" Fine answer. Then: "What about the alternatives?" The bot replied with a polite request to re-specify the question, because it had no idea what "alternatives" referred to. Each request was a clean slate: question in, answer out, nothing remembered.

The wrong turn: server sessions

My first instinct was sessions. A server-side conversation store, probably Redis, keyed by session ID, so the backend could stitch turns together. FastAPI would hand out the store handle through a dependency per request [3]. It was the rest that felt heavy: a session store means state, state means an expiry policy, and expiry policy means deciding when a half-finished conversation gets evicted. Cache-aside guidance says entries can just expire naturally [2], but I would be running new infrastructure so a blog chat can remember three questions.

Turns out the problem was never missing sessions. LLM APIs are stateless per request, and multi-turn conversation isn't the server remembering anything. The client resends the previous messages as part of each request [1]. The "memory" is an illusion maintained by whoever calls the API. If context travels with the request, it belongs to the sender, and the sender is the browser, which already has per-origin storage that survives reloads and restarts: localStorage [4].

What I actually built

The contract first. The ask endpoint grew a history field, validated by Pydantic at the trust boundary: max 5 Q&A pairs, question capped at 500 characters, answer at 2000. The frontend collects the last 3 completed turns from localStorage and ships them with every ask. Zero server state.

# use_cases.py — excerpt
history_block = _build_history_block(history or [], locale)
prompt = (
    f"{system.get(locale, system['id'])}\n\n"
    + (f"{history_block}\n\n" if history_block else "")
    + "\n\n".join(blocks)
    + f"\n\n{_ASK_QUESTION_LABELS.get(locale, 'Pertanyaan')}: {query}"
)

The backend doesn't trust the client blindly. Those caps exist because a hostile client could stuff kilobytes of fake "history" into the prompt. Of the pairs received, only the last 3 turns get injected as a "Previous conversation" block, with a hard 900-character total budget and a 300-character cap per answer. One detail I like: the validation caps are generous while the injection budget is stingy. Validation is a trust boundary; prompt budget is a cost decision. Keeping them separate means I can tighten the prompt side later without touching the API contract.

Why I prefer this over server sessions: it scales for free. Any number of app instances, any region, no coordination. If the backend restarts mid-conversation, the user's context survives because it was never on my side to lose. The trade-off is real: history is client-supplied, so a tampering user can fake a conversation. For a blog chatbot that costs nothing; they would be lying to themselves with their own token budget.

The lazy path won. The Redis sketch got deleted and the whole feature is maybe sixty lines, most of it validation. The follow-up question works now, and the server still forgets everything the moment it answers. The answer cache keeps behaving as plain cache-aside, letting entries expire naturally [2].

Sources

[1] OpenAI API Docs: Conversation state
[2] Redis Cache-aside Guide
[3] FastAPI Best Practices for Dependency Injection
[4] MDN Window.localStorage