Three Gates of Public Registries Before Writing a Scraper
Probing nine public-registry hosts: two open, one captcha, two WAF. Learn the three access-gate patterns before writing a single line of scraper.
TL;DR
Read-only probes of Jakarta court SIPP portals found mixed access: two open, one captcha-gated, two WAF-blocked, others thin or refusing. SIPP is built per court, so variation is expected; each host gets one polite probe saved as a local fixture. No captcha solving or header tricks; three hosts form the pilot, classified locally since menu tokens don't filter.
The pilot plan named four Jakarta commercial courts as fetch targets. The read-only probes flipped that set: two courts open, one serving a captcha page, two answering HTTP 403, and one thin variant without a bankruptcy menu.
The wrong assumption sounded reasonable: portals under one institution must share the same endpoint structure. If one is reachable, the rest surely are. In reality, SIPP is officially a web application built per work-unit court [5], so variation between hosts is expected behavior, not an anomaly: a thin 2.5 KB variant host without a bankruptcy menu, one that refuses connections, and a variant on another domain that blocks.
That is why mapping the access gates comes first, before a single line of scraper is written.
Run one probe per host and save the fixture
The method is cheap: one polite GET per host, with a User-Agent that states the data-collection purpose transparently. Every raw response is saved as a committed HTML fixture file, so page-structure inspection never re-hits the target server. Each host's status is recorded as measured:
| Host | Status | Note |
|---|---|---|
sipp.pn-jakartabarat.go.id, sipp.pn-jakartautara.go.id, sipp.pn-depok.go.id | open | same layout, full menu |
sipp.pn-jakartapusat.go.id | captcha | reCAPTCHA validation page |
sipp.pn-jakartaselatan.go.id, sipp.pn-tangerang.go.id | HTTP 403 | refuses browser User-Agents too |
sipp.pn-bogor.go.id | thin variant | 2.5 KB, no bankruptcy menu |
sipp.pn-bekasikota.go.id | refused | connection refused |
The command shape:
curl -A "NamaBot/1.0 (kontak: [email protected])" \
-s -o fixture.html -w "%{http_code}" \
https://daftar-hitam.inaproc.id/Probe ethics follow RFC 9309: robots rules are not a form of access authorization [3]. Politeness therefore stays a design burden: an honest User-Agent, as few requests as possible, cached fixtures, and hosts that refuse are recorded, not fought.
Learn the three gate patterns
The open pattern. Page one of the LKPP National Blacklist renders fully server-side: No, Supplier, Display Scenario, Package Number, Package, and Effective Date columns, newest sanctions on top. No pagination links exist in the raw HTML because deeper pages are JavaScript-driven; a daily snapshot of page one is enough for the pilot. This source also carries a clear legal basis: an information system holding the identities of suppliers sanctioned by procurement officials, grounded in Perpres 16/2018 [1].
The captcha pattern. The central Jakarta court serves a validate.php page carrying reCAPTCHA: a bot-protection platform that collects browser signals and returns an encrypted token for backend assessment [7]. A script client without a browser environment cannot pass, and precisely because of that the decision is firm: no captcha solving, the host is recorded as gated and removed from automated targets.
The WAF pattern. Two courts answer 403 even to browser User-Agents. A web application firewall applies a set of rules to HTTP conversations to protect the server [6]; a 403 is a server-side decision, not a client bug to fix by swapping headers or slowing requests down. Accepting it as the boundary is the cheapest move.
Classify locally, not via menu tokens
The most honest trap was found inside an open site. Per-case-type menu links use opaque tokens shaped like list_perkara/type/<token>, yet the token page does not actually filter; every case type still renders. Classification therefore happens locally: the general case-list table is parsed, the Pdt.Sus-Pailit and Pdt.Sus-PKPU case-number patterns mark bankruptcy and PKPU signals (classification text as the fallback), and company names come from the parties column with company or partnership prefixes.
The probes end in a configuration decision, not a theoretical conclusion: three hosts enter the pilot set, one is recorded captcha-gated, two recorded WAF, the rest recorded as fair variation. Every decision above can be re-read from fixtures without touching anyone's server.
Sources
- PPID LKPP: National Blacklist Application
- APScheduler Documentation: apscheduler.triggers.cron
- RFC 9309: Robots Exclusion Protocol
- JDIH BPK: Law No. 27 of 2022 on Personal Data Protection
- Supreme Court Documentation: Introduction to SIPP Backup
- OWASP: Web Application Firewall
- Google Cloud Documentation: Fraud Defense overview (reCAPTCHA)