data-engineering

A scraping workspace kept apart from the app, and a replayable single-door API delivery: resumable queue, stable dedupe key, ship log.

TL;DR: A mid-run interruption exposed the risk of writing scrape results straight into the app database, so this commit isolates all scraping tools and stores per-page, per-subtab queue rows with status and attempt logs on disk. Resuming is then deterministic. Shipping happens separately through one ingest endpoint, using stable deduplication keys and exponential backoff so idempotent retries make replays safe.

Why gated data harvests save verbatim raw text before parsing: an idempotent ledger, a resumable cursor, and retroactive rebuilds.

TL;DR: When scraping captcha-gated, session-locked court portals, save the verbatim rendered text first and parse later from local files. Resumable cursors and idempotent upserts keep crashes from corrupting data. This bronze-layer approach means parser fixes apply retroactively without ever hitting the fragile site again.