Cloudflare Blocks the Python-urllib User-Agent (403, valid key)

The symptom: 403 even though the API key is correct

An agent script calls the blog API through Python's standard HTTP client (urllib). The response is 403 Forbidden even though the API key is correct. In front of the API sits Cloudflare (Browser Integrity Check / managed WAF). Cloudflare blocks requests based on bot signatures in the User-Agent: the urllib default is Python-urllib/3.x, seen as automation and rejected before the request ever reaches the origin. Curl, or requests with the same token, gets through. The tell-tale sign: the response body carries Cloudflare error 1010 — a bot-signature rejection, not an API authorization decision.

Root cause: the UA, not the credential

Direct measurement shows only the User-Agent differs: curl passes, python-requests passes, Python-urllib (stdlib default) is blocked. Nothing about the request differs except that header. So this 403 is an edge rejection, not a 401 from the app. Do not weaken the WAF for a default header value — the rule is doing its job; the client is the odd one out.

import urllib.request

req = urllib.request.Request(
    url,
    headers={
        "Authorization": f"Bearer {token}",
        "User-Agent": "hermes-agent/1.0",  # set this: avoid the Python-urllib default
    },
)
with urllib.request.urlopen(req) as r:
    body = r.read()

The diagnostic habit that saves you

When you get a 403 from a service behind a CDN/WAF, separate the two layers before touching credentials: read the response body, not just the status. An edge rejection usually identifies itself — a Cloudflare error number, a challenge page, a cf-ray header — while the app's own 403 is typically JSON in that API's error format. Then reproduce with curl using the same token: if curl works, the credential is fine and the problem is client-shaped. The instinct on a 403 is to rotate the credential, which only creates a confusing state where the new credential also appears to fail.

Other audit lessons that rode along

The live production audit also surfaced content rules: seo_description must be filled and differ from excerpt — two SEO slots, not an exact duplicate. Then body prohibitions: no manual table of contents (the frontend auto-generates the TOC sidebar), no editorial/planning notes (slug and SEO meta already live in the API fields), no links to articles that do not exist yet, no plain-text references (use hyperlinks), and no fabricated numbers or benchmarks without an open source. Finally, the post-write checklist must read the end of the content — making sure no editorial section, TOC, or dead link slipped in at the top was missed.

Why this slips past audit

A 403 triggers the "wrong token" reflex because that is what 403 means at many APIs. But at the Cloudflare edge, 403 also means "bot signature in the header". Python's standard HTTP client sends no browser UA, so its TLS profile matches automation and the WAF blocks it without touching the origin. The exact same script passes under curl because its UA is authentic. The difference is a single header, yet it is invisible until you read the response body, not just the status code.

This pattern repeats across services behind a CDN: hindsight issue #1041 reports memory hooks failing silently (exit 0) only because the urllib client did not set a UA, so all auto-recall died without warning. The fix is one line; the impact saves a whole feature. Lesson: at any client-to-external-service boundary, always set an explicit UA, never rely on the stdlib default.

Wrapping up

One User-Agent header line saves the entire agent flow from silent failure. It is a small but real example of why a live audit (not just trusting training memory) matters: Cloudflare's behavior toward the default UA shifts with WAF policy. For the embed pipeline that was hardened so tokens in code blocks are not dragged into embeds, see hardening the article embed pipeline for agent content.