The Chat DOM Only Holds Half the Answer
Scraping an AI chat answer from the DOM gets you half of it. The full text lives behind the Copy button, plus a strict output contract.
TL;DR
The Playwright scraper only caught half of the AI answer because the chat UI uses virtualization and only renders the visible part of the DOM. The fix: set an output contract — the model must wrap its answer in one code block ending with the sentinel ===SELESAI===, with no follow-up questions. The rest is handled by polling from a second tab plus a Copy-button click to read the clipboard, so the DOM is no longer a constraint.
The scraper came back clean with 150 words of a 600-word answer. I re-ran. Same. Scrolled the chat container, waited a second, scraped again. Still 150 words. My Playwright script was eating half the AI's response, and I couldn't figure out why the DOM just refused to hold the whole thing.
Virtualization keeps the DOM half-empty
Modern chat UIs use list virtualization [2]. They only render DOM nodes for items currently visible in the viewport. Scroll up, and the older messages get their DOM nodes recycled, with the node repurposed for new content. Scroll down, same thing happens to the top. At any given moment, the full conversation doesn't exist in the DOM. Only the visible slice does.
This is great for performance, and miserable for scraping. A chat with hundreds of messages does not need thousands of DOM nodes. But it is terrible for scraping. My script was reaching for content that simply was not there. Part of the answer had already scrolled out of view, and those nodes were gone.
My first instinct was to force-scroll the container to load everything into the DOM. No effect. Virtualized lists do not load items into the DOM on scroll; they render on demand and discard whatever is off-screen. I was fighting the architecture instead of using it.
Layered fallbacks
I needed a reliable way to grab the full text. The most obvious path: click the page's Copy button, then read the clipboard [1]. This sidesteps virtualization entirely because the Copy handler has access to the full data model and never reads from the DOM. And the Clipboard API only allows reading after a transient user activation. A programmatic button click from Playwright satisfies that requirement. It's the cleanest primary path.
Why a click matters is spelled out in the spec: reading from the clipboard requires that the user recently interacted with the page, and browsers in practice layer their own permission prompts on top [1]. A Playwright click on a real button counts as real activation as far as the page is concerned, so the read goes through without fighting the browser.
The catch: not every chat UI has a Copy button. So the script got layers. The accessibility tree snapshot works as a structural anchor because screen readers see the full content even when DOM nodes are recycled. innerText as a last resort, because it at least captures what's visible, even if it's partial.
But layering fallbacks felt like treating symptoms, because the real problem sat upstream.
Contract at input, not parsing at output
The shift that actually solved this: instead of scraping whatever the AI decided to output, I defined an output contract. The model must wrap its entire answer in one code block. It must end with a sentinel marker ===SELESAI===. It must never ask follow-up questions, because a question means another turn and another scrape.
<jawaban model>
...teks lengkap...
===SELESAI===Once the contract was in place, the DOM stopped mattering. The first version polled the original tab: reload it, watch the DOM, repeat. That broke the conversation state every single time. Reloading a chat tab is not free, and observers kept tripping over the streaming UI mid-answer. A second tab pointed at the same stable chat URL turned out to be the whole trick: the original tab stays asleep, the poller works in its own quiet room, and the two never step on each other. Now I open that tab, poll for the sentinel using Playwright's auto-waiting, which surfaces a TimeoutError when the checks don't pass in time [3], and read the clipboard after the Copy button click. The scraping is finally predictable.
It is the same lesson I wrote up when a control flag got split across two SSE chunks: contract at input is cheaper than parsing at output. Define the shape of what you expect before you try to extract it. Every hour spent building a smarter scraper is an hour not spent on the one-line prompt that removes the problem.
---
Sources:
- Clipboard API (MDN Web Docs) [1]
- Virtualize large lists with react-window (web.dev) [2]
- Auto-waiting (Playwright Documentation) [3]
[1] https://developer.mozilla.org/en-US/docs/Web/API/Clipboard_API [2] https://web.dev/articles/virtualize-long-lists-react-window [3] https://playwright.dev/docs/actionability