A Failure-Proof Research Script: Two-Provider API Key Rotation

`web_search` returned 403. Not a transient error — permanent. The Firecrawl keyless route that used to work for quick searches during research sessions was just gone one day. No deprecation notice I'd seen, no gradual degradation. Just 403, every time.

I was mid-research on EYD, the Indonesian language standardization topic, and I needed search results from multiple sources fast. The keyless route dying meant I had to pivot to the actual Firecrawl API. Which meant API keys. Which meant rate limits.

One key per provider isn't enough

My first instinct was dead simple: grab a Firecrawl API key, hardcode it, done. One key, one provider, move on.

That lasted about twenty minutes of real usage. Firecrawl's `/search` endpoint costs 2 credits per 10 results [5] — fine for occasional lookups, but exploratory research across many queries burns through a free tier fast. And Tavily? It throws a 429 "excessive requests" [6], plus a 432 "exceeds your plan's set usage limit" when you actually exhaust the allowance.

So I needed more keys. But "more keys" quickly became a design question: how do you rotate through them without hardcoding or leaking them?

The answer landed in `web-direct.py` — 125 lines in `hermes/scripts/`, handling search, scrape, extract, and health checks. It reads `FIRECRAWL_API_KEYS` and `TAVILY_API_KEYS` from environment variables, both comma-separated. When a request fails, it doesn't just jump to the next provider. **It tries the next key within the same provider first.** Only when all Firecrawl keys are exhausted does it move to Tavily. Two-layer rotation.

This matters because a 429 on key #1 doesn't mean your Firecrawl account is burned. It means that particular key hit its rate window. Key #2 might be totally fine. Jumping straight to Tavily wastes a credit there when you still have Firecrawl capacity sitting behind a different key. The script never prints keys either — they stay in env vars and that's it.

The ISP that was lying about DNS

This part took longer to figure out.

Some domains I was researching kept resolving to wrong IPs. Not "moved to a new server" wrong — "doesn't exist anymore" wrong. The Indonesian Ministry of Education's spelling dictionary site, `ejaan.kemdikbud.go.id`, had migrated to `ejaan.kemendikdasmen.go.id`. My ISP's DNS resolver was still returning the old IP for the old domain, so `curl` connected to a server that no longer served the content I expected. The requests succeeded. The content was just... irrelevant.

I caught it because `web-direct.py` runs a health check using DNS over HTTPS against Cloudflare's resolver [7]. The JSON response came back with `Status: 3` — which is NXDOMAIN [8]. The domain genuinely doesn't exist anymore according to global DNS, but my local resolver was feeding me stale data. Or worse, fabricated data.

The fix: `curl --resolve` to force a specific IP, bypassing the poisoned resolver entirely [9]. In practice it's `curl --resolve host:443:IP`. Ugly, but when your ISP's DNS lies, you stop trusting it.

DoH as a verification layer — not the primary resolver, but a lie detector — turned out to be the most practical pattern. The script queries DoH to check if a domain is actually alive, compares against what the local resolver says, and uses `--resolve` when they disagree. Not elegant. Effective.

Why this pattern keeps repeating

I run the same principle across my tooling; the AI agent skills research stack article covers that side. The whole script rests on one assumption: free API tiers will throttle you, DNS resolvers will lie to you, and single points of failure hide everywhere.

Two-layer key rotation accepts that rate limits are per-key, not per-account. DoH health checks accept that network infrastructure between you and the internet isn't trustworthy. Comma-separated env vars mean you add a new key without touching the script.

125 lines enough? For a local research tool that needs to search, scrape, and extract without dying on the third API call — yeah. Could add exponential backoff, circuit breakers, persistent caching. But it's a script I run locally for research. **It just needs to not fail silently when Firecrawl's free tier says "enough."**

---

Sources:

• Source 5 [5]
• Source 6 [6]
• Source 7 [7]
• Source 8 [8]
• Source 9 [9]