devops

A 429 rate limit froze my research pipeline. Designing multi-key rotation: per-HTTP-status cooldown, single-pass failover, and errors that should not rotate.

TL;DR: The pipeline hit 429 rate limits, so the author added key rotation with per-status cooldowns: 60s for 429, 300s for 402, an hour for auth errors. Rotation is single-pass to avoid infinite loops, and request errors like 404 throw immediately. The author stresses rotation only raises ceilings; fix wasteful requests first.

Copying an active SQLite file is not a backup. Lessons from a light backup script: hot backup, verified atomic tar, and layered pruning.

TL;DR: The script backs up SQLite databases using the official .backup API instead of raw cp, since copying active WAL files can yield corrupt snapshots. Archives are staged, verified with tar, then atomically renamed, with flock preventing overlap. Pruning keeps 14 days plus a floor of 3 newest archives, so failures can't silently destroy valid backups.

Migrating the Hermes watchdog to host-native was not just swapping paths. Lessons in git diff --quiet, set -euo pipefail, and a zero-token pattern.

TL;DR: Migrating Hermes to host-native broke hermes-patch-watchdog.sh, which still pointed at the old container path. The fix required more than a find-and-replace: git semantics shifted, the venv moved, and set -euo pipefail turned silent mid-script failures into false healthy signals. Every check now fails loudly, since for a silence-means-healthy watchdog, noise beats quiet mistrust.

The fallback fired, the article shipped, and the real failure was one clipboard probe returning 410 Gone. Diagnosis before fallback, now a written rule.

TL;DR: A clipboard probe returned HTTP 410, exposing that an automated fallback had silently routed around a Qwen generation failure. The fallback fired on exit code 1 without diagnosing the cause, so the pipeline looked green while the primary path rotted. Now every failure triggers a doctor script first, and exit codes are disambiguated.

An hourly wave of 504s looked like an attack. The logs pointed at a dormant rate limit and a PHP-FPM pool out of workers.

TL;DR: Hourly 504 waves looked like a DDoS but were actually PHP-FPM worker exhaustion, worsened by request_terminate_timeout killing slow requests. Nginx rate limiting never fired because Cloudflare hid real client IPs, collapsing everyone into one $binary_remote_addr bucket. Fixes: enable Cloudflare caching, tune PHP-FPM pool settings, and rotate container logs with copytruncate.

A crashed cron loop resent the same prompt. A send-then-poll async contract plus two state files make Qwen chat generation safe to re-execute.

TL;DR: After a cron crash resent duplicate prompts and burned quota, the author swapped queue infrastructure for two empty files as idempotency keys: one marks the prompt sent, the other the result done. A four-state exit code contract (42, 3, 0, 1) mirrors the async request-reply pattern client-side. Since cron retries crashed jobs, making re-runs harmless beats a database or broker.

Months spent blaming an anti-bot; the culprit was an Indonesian ISP's DNS redirecting duckduckgo.com to Trustpositif.

TL;DR: Headless browser tests inside Docker kept failing on DuckDuckGo, and I wasted months blaming anti-bot fingerprinting. The real culprit was Indonesian ISP DNS poisoning; Kominfo's 2015 mandate redirects port 53 queries, so duckduckgo.com resolved to a wrong IP. The fix was one flag: docker run --dns 1.1.1.1 restored access instantly.

Free Qwen chat via Playwright is now the primary draft engine for my article cron, with a local router9 fallback wired like a circuit breaker.

TL;DR: The author scrapes Qwen's free web chat via Playwright, copying answers from the clipboard since virtualized DOM rendering hides long replies. Knowing free tiers die without notice, a circuit breaker routes to local router9 after two failures, giving genuinely separate failure domains. The bet isn't that scraping lasts, just that the pipeline keeps running when it doesn't.

A two-line .gitignore whitelist commit turned into a full backup redesign: Git repo, restic, immutable cloud, 3-2-1-1-0 without a NAS.

TL;DR: A two-line .gitignore change led me to audit my backups, and I realized my NAS setup was untested and ransomware-vulnerable. I replaced it with a 3-2-1-1-0 approach: git bundles, restic snapshots, and immutable cloud storage. Monthly restore drills now verify the whole thing actually works.