tooling

Pagination URLs carry rotating session tokens that die within the hour. Drive the app's own search form instead, then flatten parsed values before comparing.

TL;DR: Apps that mint session-bound URLs make deep links unreliable, so drive the search form instead: type the exact document number, verify the detail page, and log the query rather than the link. Normalize parser output by flattening lists and lowercasing strings before status comparisons. Flag mismatches for human review instead of stopping the run.

A scraping workspace kept apart from the app, and a replayable single-door API delivery: resumable queue, stable dedupe key, ship log.

TL;DR: A mid-run interruption exposed the risk of writing scrape results straight into the app database, so this commit isolates all scraping tools and stores per-page, per-subtab queue rows with status and attempt logs on disk. Resuming is then deterministic. Shipping happens separately through one ingest endpoint, using stable deduplication keys and exponential backoff so idempotent retries make replays safe.

Why gated data harvests save verbatim raw text before parsing: an idempotent ledger, a resumable cursor, and retroactive rebuilds.

TL;DR: When scraping captcha-gated, session-locked court portals, save the verbatim rendered text first and parse later from local files. Resumable cursors and idempotent upserts keep crashes from corrupting data. This bronze-layer approach means parser fixes apply retroactively without ever hitting the fragile site again.

The preview panel session is the scraping credential: rotating URL tokens, cosmetic click failures, CDP left dead.

TL;DR: When scraping Jakarta's court portal, external fetches only hit captcha walls because the browser session inside the panel carries the real credentials. Rotating tokens force click-driven navigation, and automation success flags lie, so verify via URL and DOM instead of auto-retrying. This approach harvested 4,828 bankruptcy cases across 242 pages, keeping sessions in-panel with checkpoints instead of URL replays.

Finding a site's own GraphQL endpoint in its public JS bundle: 5330 rows in 7 requests, not 533 HTML pages.

TL;DR: Expecting a tedious HTML scraping job for Indonesia's procurement blacklist, the author found an unauthenticated GraphQL endpoint exposed by the site's XLSX export. A perPage of 800 meant the full 5,330-row dataset downloaded in just seven requests and about ten seconds. The daily snapshot plan was retired in favor of a simple full-dataset sync.

Stealth proxies beat WAF walls and die at browser-measuring captcha gates.

TL;DR: Stealth proxies hit captcha walls, not WAFs: Jakarta Selatan hid Cloudflare Turnstile, Jakarta Pusat ran reCAPTCHA v2. Real browsers cleared both—Chromium passed Turnstile untouched; a Firefox anti-detect profile solved reCAPTCHA with Tab plus Space, no solver service. Detail pages need live sessions, so the build now tiers access: proxies for WAFs, browsers for captchas, supervised runs for Jakarta Pusat.

Probing nine public-registry hosts: two open, one captcha, two WAF. Learn the three access-gate patterns before writing a single line of scraper.

TL;DR: Read-only probes of Jakarta court SIPP portals found mixed access: two open, one captcha-gated, two WAF-blocked, others thin or refusing. SIPP is built per court, so variation is expected; each host gets one polite probe saved as a local fixture. No captcha solving or header tricks; three hosts form the pilot, classified locally since menu tokens don't filter.

Two registries passed probing: scrapers behind one seam, idempotency forced via content hash, and a daily 05:00 ingest with jitter.

TL;DR: A pilot ingests daily signals from the LKPP procurement blacklist and three courts' SIPP bankruptcy/PKPU case lists through a single SignalSource interface. Idempotency comes from content hashes and unique constraints, so re-runs never duplicate data, and one court failing doesn't kill the run. The job runs once daily via APScheduler under strict politeness rules, with privacy safeguards built in.

Building scrapers from endpoint guesses fails fast: a census of 348 SIPP endpoints and 12 official sources sets the foundation before the first crawler runs.

TL;DR: A census of 350 Indonesian district courts found 348 active SIPP endpoints, confirming no central API exists. Twelve other official sources were tiered by verification, with the LKPP blacklist pilot first, then a phased per-court SIPP crawl. The real value comes from reading all sources as one curve covering pre-distress, distress, and post-collapse signals.