Exit Code 1, a 410, and a Fallback That Lied
The fallback fired, the article shipped, and the real failure was one clipboard probe returning 410 Gone. Diagnosis before fallback, now a written rule.
TL;DR
A clipboard probe returned HTTP 410, exposing that an automated fallback had silently routed around a Qwen generation failure. The fallback fired on exit code 1 without diagnosing the cause, so the pipeline looked green while the primary path rotted. Now every failure triggers a doctor script first, and exit codes are disambiguated.
Exit Code 1, a 410, and a Fallback That Lied
camofox-doctor.py runs its checks in layers: L0 health, L1 container hardening, L2 smoke REST, L3 Qwen login profile, L4 clipboard probe. That morning L4 came back with one line I did not expect: HTTP 410 Gone. Not a timeout, not an auth failure. Gone. The server was telling me, politely, that the resource the clipboard probe hits no longer exists.
Here is the embarrassing part. Qwen itself was fine.
What actually happened that morning
Hours earlier, a cron cycle kicked off article generation through Qwen. The process exited with code 1. My automation did what it was designed to do: it fell straight back to router9, published the article, and reported success. No alert, no log entry worth reading, nothing. From the outside, the pipeline was green.
Because the fallback worked, nobody looked. Including me. I only found the real failure because I happened to run the doctor script later that day, and the doctor is loud where the fallback is quiet.
So the whole incident was a tiny clipboard probe hitting a resource the server explicitly says is gone. One endpoint. A 410. And an automation layer that treated "primary failed" and "primary failed for a reason worth understanding" as the same event.
A circuit breaker in reverse
The Circuit Breaker pattern says that when failures hit a threshold, you temporarily stop calling the failing resource instead of hammering an operation that is unlikely to succeed [2]. Microsoft's own wording: an application shouldn't continually retry an operation that's likely to fail [2]. The Retry pattern is the complement, and it only makes sense for transient, self-correcting faults, where a retry after a suitable delay will probably work [3].
My fallback was neither of those. Think about what it actually did:
- No diagnosis of why the primary failed
- No state carried forward about the failure
- No new information gained before choosing a path
A circuit breaker gathers failure counts, then opens. A retry waits, then tries again. My fallback skipped both halves. It saw exit code 1 and routed around the problem instantly, which is the circuit breaker pattern run backwards: instead of blocking traffic to a sick resource until it recovers, it quietly rewarded the sickness with a workaround.
I used to think fallbacks were free insurance. I don't anymore. A fallback with no diagnosis is not insurance, it is amnesia with a nice interface. Every time it fires, the root cause gets one layer deeper, because the visible symptom (missing article) never appears. The system can run for weeks like this, "working" the entire time, while the primary path rots.
Restarting was never the fix
My first instinct when something in a container misbehaves is the same as everyone's: restart it. This time I stopped to check whether that would even do anything, and the Docker docs are blunt about it. A restart just shuts down the container and starts it again, which resets runtime state only [1]. Files written inside the container live on a writable container layer, and volumes persist even after the container itself is removed [4].
Translation: if the broken thing is a stale file, a changed endpoint, or a bad login profile sitting on the writable layer or a volume, restarting is theater. The broken state survives the reboot. That is exactly why the doctor script checks in layers instead of just pinging the container. L0 through L2 tell you the box is alive. L3 and L4 tell you whether the thing inside the box still works. The 410 was never going to be fixed by docker restart, and fixing the broken layer was the real fix.
The new rule
The pipeline now works like this: any exit 1 from Qwen generation requires a full camofox-doctor.py run first. Fallback to router9 is allowed only when the doctor reports ready. I also untangled the exit codes, because "1" had been doing too many jobs. Exit 42 means the job was queued, exit 3 means generation is still processing, and only exit 1 is a real failure. Three meanings hiding in one number is how you end up routing a "still working on it" into an emergency path.
The rule I'd offer from this: whenever you build a fallback path, ask whether the system checks why the primary failed before it jumps to the secondary. If the answer is no, you have not built resilience. You have built a bug that hides other bugs, and it will run unattended.
That is the kind of feature I no longer ship on purpose.
Sumber:
1. [docker container restart](https://docs.docker.com/reference/cli/docker/container/restart)
2. [Circuit Breaker pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker)
3. [Retry pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/retry)
4. [Docker storage overview](https://docs.docker.com/engine/storage/)
Sources:
- Docker Docs, docker container restart [1]
- Microsoft Learn, Circuit Breaker pattern [2]
- Microsoft Learn, Retry pattern [3]
- Docker Docs, Storage overview [4]
[1] https://docs.docker.com/reference/cli/docker/container/restart [2] https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker [3] https://learn.microsoft.com/en-us/azure/architecture/patterns/retry [4] https://docs.docker.com/engine/storage/