AI Chat Scraper Audit: From False Positives to Precise Regex
A loose completion gate called a half-written answer done. One audit produced four fixes: pre-answer failure patterns, an anchored heading gate, probe-copy.
TL;DR
The scraper kept cutting AI answers short because loose completion gates matched "Sources" inside citation lines. The fix narrows trust: failure patterns only count before any answer text, and the completion gate requires a contracted final heading via a strict regex. Added a --probe-copy diagnostic and evidence logging; selftest grew from 72 to 79 asserts.
I was monitoring the logs of chat-research.py in the github.com/didiet86/my-apps repository when the execution abruptly cut short. The scraper declared the answer complete while the AI model was only halfway through writing. I initially guessed this was due to failure patterns like "rate limit" or "login" being detected mid-text. Turns out, my guess was completely wrong. The word "Sources" inside a regular citation line was being treated as an end signal by a loose completion gate. The scraper grabs answers from two places, the Copy-button clipboard and the page text itself, and both paths fed this one polling loop. Commit 2f42ccd reworked that loop end to end, and every fix pointed the same direction: narrower trust, not smarter guessing.
The Trap of Failure Patterns Mid-Content
The first issue lay in failure pattern detection. Previously, the scraper would instantly abort the session upon spotting keywords like "Cloudflare" or "password". After the audit, I realized that once the model starts answering, the page text contains the answer prose. Those keywords might just be part of the content itself, not the session state.
The solution is strict. Failure pattern detection is now trusted only before any answer text exists. The trade-off is clear. If a real mid-stream logout occurs, the system will only detect it at the --timeout deadline. This is a fail-soft compromise far safer than prematurely truncating valid responses. I also integrated the free-tier browser automation flow 2 and clipboard answer recovery mechanisms 3 to ensure stable execution without heavy overhead.
Explicit Output Contracts Beat Loose Heuristics
The second error stemmed from an overly permissive completion gate. Previously, any substring "Sources" or any markdown heading was considered a finish line. This frequently trapped the scraper on half-baked answers that happened to have an interim heading.
The fix demands precision. The completion gate now requires the contracted final heading to appear as its own line. I implemented the DONE_HEAD_RE regex ^#{1,3}\s+(?:Sumber|Sources)\s*:?\s*$ with the re.M MULTILINE flag [1]. In MULTILINE mode the caret matches at the start of the string and right after each newline. This aligns perfectly with the CommonMark ATX headings specification, which requires 1 to 6 # characters followed by a space or end of line [3]. This approach keeps the scraper from being fooled by citation lines.
Two Extraction Paths, Two Rule Sets
The audit also exposed the need for sharper diagnostics, so I added an opt-in --probe-copy diagnostic that sends one test message containing a fenced code block. The system then compares the clipboard candidate length against the page. If the clipboard reaches 60 percent of the page length and still retains a code fence, it writes a single evidence line to evidence.jsonl [4]. This JSON Lines format is highly convenient for log files since it is newline-delimited.
And UI_JUNK strings on the clipboard path are now merely a log-only warning. Clipboard content is message-scoped quoted prose, not SPA chrome. The page path still rejects them hard. The Python logging module [2] allows every module to participate via getLogger, so app logs and library logs land in one stream. Thanks to these changes, the selftest grew from 72 to 79 asserts 1.
This experience taught me one thing. Trying to expand failure detection often just adds noise. Narrowing detection to the right moment and demanding explicit output contracts is the real key. Reproducible evidence files ultimately rescue debugging sessions that once seemed completely blocked. None of this is exotic: a regex with the right flags, a warning instead of a rejection, one diagnostic that writes its own log line. The scraper never gets to declare victory on a loose match again; the contract says when it is done, and the evidence file says what happened.