When Deep Links Rotate, the Search Box Is the Address
Pagination URLs carry rotating session tokens that die within the hour. Drive the app's own search form instead, then flatten parsed values before comparing.
TL;DR
Apps that mint session-bound URLs make deep links unreliable, so drive the search form instead: type the exact document number, verify the detail page, and log the query rather than the link. Normalize parser output by flattening lists and lowercasing strings before status comparisons. Flag mismatches for human review instead of stopping the run.
Recognize Session-Minted URLs
During a routine data harvest for the fictional DemandScope project, a pagination URL appeared containing a lengthy, rotating base64 token. The token changed with every page transition, embedding session-specific routing data directly into the address. Attempting to fetch one of these URLs in a later automated session failed immediately, because the session context that minted the token no longer exists on the server. The detail page URL itself stays constant, yet its functional identity lives entirely in the active session state rather than in the URL string.
HTTP is stateless: each request and response pair operates independently, and the session ID binds credentials and access controls to that traffic [1]. The W3C doctrine states that a cool URI does not change, and that URI stability is the publisher's duty, not the linker's [2]. The target application cannot meet that standard. Once an app mints unstable deep links, the practical move is to stop constructing URLs altogether.
Take the Address from the Search Box
Drive the application's own search form instead of reverse-engineering its fragile links. An HTML form is, at its core, a user-friendly way to configure an HTTP request [3]. The search box becomes the one address the application guarantees to translate consistently for any given query, no matter what the session layer does underneath. The method demands discipline in recording flow rather than storing absolute links that expire.
The failure symptom is specific. If pressing Enter brings back the list page, an empty result, or a different document, the query does not exactly match the official document number. Such numbers follow a rigid format: slashes, a year, sometimes a letter block. One extra space is enough to miss. Copy the number exactly as the list row shows it, punctuation included.
The search box is not the only element with this property; many apps ship an advanced filter or a lookup menu that behaves the same way. The principle is identical: find the interface element the application itself controls, and make it the standard address in the work queue.
Three checks keep the lane honest. First, confirm the detail page renders after typing the document number and pressing Enter; its content must match the row you searched. Second, save the rendered text as-is for provenance before any parsing touches it. Third, log the flow in the ledger, not the absolute URL. The URL field records "via search box <query>", which stays reproducible long after the session token dies.
Here is a generalized bookkeeping function for a production harvest:
def harvest_via_search(query, target):
queue.add(query)
form = target.find_form("search")
form.fill("document_id", query)
form.submit()
detail = target.wait_for_detail()
save_raw(detail.text)
ledger.log(f"via search box {query}")Flatten Values Before Comparing
The second lesson from the same change concerns validation of parser output. A tolerant parser decides value types from the visual structure it reads. A status cell spanning two lines in the list can parse as a list, not a single string. The cross-check between the list status and the detail status then fails on a type difference before any real comparison happens.
Sound input validation means checking syntax and semantics, using a maintained parser, handling parsing failures, and validating values before business processing [4]. Normalization therefore belongs exactly at the comparison boundary.
The two-line fix checks the type and joins when needed. Normalizing both sides into lowercase strings before the comparison removes false failures caused by parsing quirks. The mismatch warning deliberately does not stop the run; it flags the row for human review before anyone uses the data.
list_status = " ".join(list_status) if isinstance(list_status, list) else list_status
if list_status.lower() != detail_status.lower():
ledger.warn("status mismatch list vs detail")