browser-automation

The preview panel session is the scraping credential: rotating URL tokens, cosmetic click failures, CDP left dead.

TL;DR: When scraping Jakarta's court portal, external fetches only hit captcha walls because the browser session inside the panel carries the real credentials. Rotating tokens force click-driven navigation, and automation success flags lie, so verify via URL and DOM instead of auto-retrying. This approach harvested 4,828 bankruptcy cases across 242 pages, keeping sessions in-panel with checkpoints instead of URL replays.

Stealth proxies beat WAF walls and die at browser-measuring captcha gates.

TL;DR: Stealth proxies hit captcha walls, not WAFs: Jakarta Selatan hid Cloudflare Turnstile, Jakarta Pusat ran reCAPTCHA v2. Real browsers cleared both—Chromium passed Turnstile untouched; a Firefox anti-detect profile solved reCAPTCHA with Tab plus Space, no solver service. Detail pages need live sessions, so the build now tiers access: proxies for WAFs, browsers for captchas, supervised runs for Jakarta Pusat.

Building a free article draft generator from chat.qwen.ai via automated browser: mechanics, pitfalls, and fallback that make it cron-worthy.