One API Key Just Stops Research at 429
A 429 rate limit froze my research pipeline. Designing multi-key rotation: per-HTTP-status cooldown, single-pass failover, and errors that should not rotate.
TL;DR
The pipeline hit 429 rate limits, so the author added key rotation with per-status cooldowns: 60s for 429, 300s for 402, an hour for auth errors. Rotation is single-pass to avoid infinite loops, and request errors like 404 throw immediately. The author stresses rotation only raises ceilings; fix wasteful requests first.
My article research pipeline leans on one external service to fetch page contents. One afternoon, every job stopped dead with 429 responses from the server. Rate limit. My first guess was shallow: add a delay between requests, problem solved. A delay only pushes back the clock until everything jams again. One key means one limit, and the limit gets hit exactly when you need the service most.
Different Cooldowns for Different Sins
The vendor's answer for 429 is clear: back off, and if a Retry-After header exists, honor its duration [8]. Their error table also shows that 402 is about exhausted credits and is marked non-retryable [8]. Two very different meanings. A rate limit may recover in seconds to minutes; exhausted credits will not heal by retrying five minutes later.
That is why my patch does not use a uniform wait. The _RotatingKeyFirecrawlClient class in plugins/web/firecrawl/provider.py keeps a different cooldown per status: 429 is noted for 60 seconds, 402 for 300 seconds, 401 and 403 for an hour, while network errors only get 30 seconds. These numbers are design decisions, not vendor recommendations; from the vendor came only the principle of honoring Retry-After [8].
The exception list is the part I like most. Errors 400, 404, and 422 are thrown immediately without rotation. The logic is simple: those are errors about the request, not about the key. Swapping keys for a broken payload just moves the problem. Telling what deserves rotation from what does not is the difference between failover and problem relocation.
An Unbounded Loop Is a Sweet Hang
The first design that comes to mind: loop until you find a healthy key. There is a subtle trap there. When every key is in cooldown, the key picker can keep returning the same key forever, and the loop never finishes. The process lives, logs keep turning, no progress happens. So the rotation is single-pass: each key gets tried exactly once per rotation cycle. After that, the last exception is raised and the caller decides. Failing fast is more honest than spinning without direction.
How to run it: put a comma-separated key list into the FIRECRAWL_API_KEYS environment variable. The old single-key path is untouched. Rotation state, the last active key plus cooldown notes, is guarded by threading.Lock, the most basic locking primitive in Python [10]. httpx's HTTPStatusError, raised when a response returns a 4xx or 5xx status from raise_for_status [9], becomes the signal to set a cooldown; everything else in the HTTPError family [9] counts as a network hiccup.
One more layer makes this patch durable: it is idempotent. Before touching the file it checks whether the marker already exists. Every replacement anchors on exactly one spot that must occur exactly once in the original file, and the result is compiled to make sure no syntax broke. A patch that cannot run twice without damage is a time bomb you built yourself.
My opinion here is blunt: key rotation is not a replacement for proper backoff, only a raised ceiling. A wasteful request pattern stays wasteful, now spread across several wallets. Clean up how the system asks first, and only then does rotation deserve to be called a safety net.