The Context Guard: Validating What Your Pipeline Sends
A prompt assembly failure can send a naked question to an AI chat and still pass as grounded synthesis. A cheap input-side guard keeps the chain honest.
TL;DR
The pipeline sent a naked question to the AI with zero context, yet the generic answer passed all validation because checks only ran on output, never input. The fix is a cheap context_present() gate requiring either 400 characters or a URL, tagging results as missing or ok. It warns without blocking, since harsh guards get deleted.
2 AM, and the terminal showed a clean run: the research chat chain finished, the answer file was saved, the result JSON said ok: true. Not a single error line. Then I opened the prompt that had just been sent to the AI chat: one naked question, no URLs, no ledger excerpts, no context at all. The pipeline had assembled its context silently wrong, and the generic answer that came back was still processed as legitimate research synthesis. The hole was not in the answer. It was in the submission.
My first guess pointed at the model. Weak engine, ordinary hallucination, take your pick. Wrong. Commit e283418 found a simpler and more accurate cause: the pipeline never checks what it sends. Every check runs in the opposite direction. The answer gets a minimum length check, a sentinel check, failure-pattern checks. The prompt going in? It walks straight through.
One-way validation is the trap
Two weeks ago the same tool, tools/chat-research.py, got fixed on the output side: failure keywords are only trusted before an answer exists, and the done gate demands an anchored ## Sources heading (full story here). Neat at the back of the pipe, but the chain still leaked at the front. Reading an answer assembled from a naked question is like approving a report from a staffer who was never given a brief. The problem is not model quality. It is the flow: the artifact being sent needs the same gate as the artifact being received. Anthropic's own engineering notes treat programmatic checks between chain steps as the way to keep the process on track[1].
A presence check, not a truth check
The fix is deliberately boring. A function called context_present() tests two things: the prompt is at least MIN_PROMPT_LEN = 400 characters long, or it contains an http(s) URL. Both absent means context absent. No semantic analysis, no judge model; the OR-threshold was picked precisely because it is cheap and almost never rejects legitimate intent.
The result of the check is equally low-key. Context missing? The script still runs, writes a warning to stderr, and attaches a "context": "missing" tag to the result JSON, on both the success and failure paths. With context, the tag reads "context": "ok". The principle is fail-soft: the exit code never changes, because running a manual side query on a narrow question is perfectly legitimate. The cron job is the thing that needs a light, and anyone consuming the JSON can read it. Whether to continue or stop is the caller's decision, not the guard's.
Warn, don't block
The temptation to make the guard harsh is real: anomaly found, script stops. In multi-stage automation that is the recipe for getting your monitoring switched off. Google's SRE book has said it for years: alerts that arrive without clear context train humans to ignore them, until the page that matters slips through too[2]. A gate that refuses legitimate operator intent will meet the same fate: worked around, loosened, or deleted.
For judging the detector itself I borrowed the SRE workbook's frame: precision, how many alarms actually matter, against recall, how many important events get caught[3]. The OR-threshold in context_present() is deliberately recall-first. A false positive costs one warning line; a false negative costs a whole research cycle that never notices it is producing noise. Those are not symmetric losses.
One thing I deliberately left out: content judgment. The guard does not care whether the question is good or dumb, only whether it stands on any context at all. That boundary keeps the function explainable in one sentence and its behavior predictable.
The change closes with five new asserts in the selftest, 64 to 69: short prompt, empty prompt, the 400-character floor, plus http and https URLs. Small, but the pattern travels: in a pipeline that assembles prompts programmatically, validating the artifact you send is part of the contract, not decoration. Every chain result now carries its own honesty label. A generic answer to a naked question no longer slips through silently; it walks out with the light on.