Prompt labels leaked into my AI chat: sanitize both ends
A RINGKAS label leaked into the chat bubble. The fix was not a stricter prompt but sanitizing input and output at both ends of the pipeline.
TL;DR
Label RINGKAS bocor ke UI karena prompt saja tak cukup andal membatasi format keluaran LLM. Solusinya menutup dua pintu: sanitasi teks eksternal sebelum masuk prompt, lalu render jawaban lewat react-markdown plus rehype-sanitize dengan allowlist HTML. Pelajarannya, kendali format sebaiknya ada di kode, bukan sekadar daftar larangan yang terus ditambah.
The internal chat on my blog suddenly started showing the literal text **RINGKAS:** inside the answer bubble. No error, no crash. Just a label from the prompt leaking through to the user, and worse, sometimes it arrived already wrapped in markdown bold. In the expanded results panel its twin, PENJELASAN:, had shown up too.
My first guess was predictable: the prompt was not strict enough. Add another prohibition, spell the format out again, repeat until satisfied. I added prohibitions several times and the leak kept happening. The model simply does not follow formatting instructions one hundred percent of the time. Sometimes it wraps the label in asterisks itself, as if "helpfully" tidying the output. The more prohibitions I stacked into the prompt, the more creative the violations got.
After a few failed rounds I stopped and asked a different question: what text actually flows into this prompt?
Two doors for untrusted text
The chatbot uses a RAG pattern. Content from a knowledge base gets fetched, merged into the prompt, sent to the LLM, and the answer is rendered in the UI. That means there are two points where text I did not write can enter: the knowledge base content itself, and the LLM output that does not always respect the format. Sanitizing the prompt alone only closes one door.
I closed the first door in the backend. A function called _sanitize_injected_text in use_cases.py runs before knowledge base text and CTA messages enter the prompt. It is deliberately simple: cut the text at 4000 characters, strip control characters, and remove common injection patterns like "ignore previous instructions". This scenario is called indirect prompt injection, and OWASP lists it as the number one risk in their 2025 LLM Top 10: content from an external source can alter model behavior without a human ever typing an instruction directly [3]. What convinced me to add this layer is OWASP's own note that RAG and fine-tuning do not fully mitigate the vulnerability [3]. Trusting that "my context is curated" is not a strategy.
The second door is on the output side. The old prompt banned markdown entirely; I flipped that around. Lightweight markdown is now explicitly allowed, but the person deciding which HTML elements reach the browser is me, not the model. LLM answers now render through a ChatMarkdown component built on react-markdown plus remark-gfm plus rehype-sanitize. react-markdown is safe by default because it never uses dangerouslySetInnerHTML [1]. rehype-sanitize acts as the final filter: anything not in the allowlist schema gets removed, and its defaults mirror what GitHub does [2]. The model still occasionally writes **RINGKAS:**, but whether that becomes bold text or gets thrown away is now decided by my code, not by accident.
From prohibitions to an allowlist
The prompt change itself was small but shaped differently. Before: "no markdown formatting at all", with the prohibition list growing after every incident. Now: lightweight markdown is allowed explicitly, bold for key terms and bullet lists for enumerations, everything else banned, headings and code blocks first. The difference is noticeable. A model given an official channel for formatting seems to stop inventing its own formatting outside the permission.
Labels that still slip through get cleaned at render time by a tolerant regex that matches both the bare label and the bold-wrapped variant. Not elegant, but it is the second seatbelt after the main sanitization.
The final behavior is checked by a sanitization suite running 148 tests in CI. Nothing exotic, just content tests asserting that dangerous patterns get filtered and scaffolding labels never reach the UI.
The decision I took away from this incident: stop relying on the prompt as the only formatting fence. The prompt stays, but only to give rough direction. Reliable control lives at the input, sanitizing external text before it enters the prompt, and at the output, through a renderer that only allows allowlisted elements. The alternative I threw away: adding yet another prohibition to the prompt every time a new leak appeared. In my experience, the longer the prohibition list in a prompt, the more often it gets violated.
Sources
- [1] react-markdown — official README (accessed 2026-09-01)
- [2] rehype-sanitize — official README (accessed 2026-09-01)
- [3] OWASP Gen AI Security Project: LLM01:2025 Prompt Injection (accessed 2026-09-01)