Not Just SQL: When API Fan-out Kills Your Monitor
Fixing an N+1 API fan-out in commitcheck: one pagination pass replaces 250 requests, SSL timeout gone, runtime down to 27.5s.
TL;DR
The monitor hung for over 100 seconds because each candidate triggered full pagination, creating roughly 250 requests per run. Replacing per-candidate checks with a single paginated fetch into a commit-to-slug map removed the N+1 overhead. Runtime dropped to 27.5 seconds with only a few requests and no change to the API contract.
The monitor that suddenly hung
I opened the 15-minute monitor log and found a single line: raw SSL read traceback at articles.py:1641. The commitcheck script that usually finished in 20 seconds had hung for over a hundred seconds. My first guess was wrong, I thought it was a transient network hiccup or GitHub rate limiting.
I tried a manual curl to the articles endpoint with per_page=100, it came back healthy in 1.5 seconds. I ran commitcheck locally with a longer timeout, it did finish but took almost two minutes. Not the network. It was a pattern.
One pagination replaces 250 requests
We all know N+1 in ORMs, but the same pattern happens at the HTTP layer. The old _covered_by_article(base, key, sha) was called inside the candidate loop. Every call re-paginated the whole list. The math hurts: about 62 candidates per tick times about 4 pages is roughly 250 HTTPS calls per run [2]. Each request carries DNS, TLS handshake, and latency, the monitor eventually timed out on the SSL read.
The industry guidance is explicit. AIP-158 requires pagination from the outset because delaying it is a backwards-incompatible change, default 50 can silently cut a collection of 75 [1]. GitHub only returns a default subset of 30 issues even when 1600 exist and forces navigation through the Link header [2]. Repeating full pagination per item just burns resources.
The laziest correct fix: paginate once at the start, then do membership tests. I replaced _covered_by_article with _covered_map(base, key) returning dict[str, str] of {source_commit: slug}. At the call site it is just sha not in covered_map. Navigation still respects the HTTP Link header with rel next to advance pages [3]. The contract stays identical, JSON shape, exit codes, and cursor unchanged.
# BEFORE — N+1 fan-out (paginate per candidate)
def _covered_by_article(base: str, key: str, sha: str):
page = 1
while True:
code, resp = api(base, key, "GET", f"/articles?per_page=100&page={page}")
for x in resp["data"]:
if (x.get("meta") or {}).get("source_commit") == sha:
return x["slug"]
if page >= resp["meta"]["total_pages"]:
return None
page += 1
# AFTER — single pass
def _covered_map(base: str, key: str) -> dict[str, str]:
out: dict[str, str] = {}
page = 1
while True:
code, resp = api(base, key, "GET", f"/articles?per_page=100&page={page}")
for x in resp["data"]:
m = (x.get("meta") or {}).get("source_commit")
if m:
out[m] = x["slug"]
if page >= resp["meta"]["total_pages"]:
return out
page += 1
GitHub also documents that the maximum per_page is 100 and larger values are silently coerced to 100 without an error [2]. So 100 per page is the intentional ceiling. Page tokens themselves must be opaque and URL-safe so users cannot deconstruct them [1].
Results I can verify myself
After the patch I measured again. From hanging for over 100 seconds to 27.5 seconds. From roughly 250 requests per tick to just a handful to build the map. I deliberately did not raise the monitor timeout, raising the timeout only hides the smell.
I prefer this one-time mapping over network-level caching. Caching helps, but the root cause was the loop. If you ever see logs with repeating requests that differ only in page=, try turning it into a single fetch up front. Lazier, faster.
Sources
[1] google.aip.dev/158
[2] docs.github.com — Using pagination in the REST API
[3] developer.mozilla.org — Link header