I finally split my transcription failures into two families
My transcription pipeline died silently mid-window. A hang turns out to be an exception that must be caught, and every failure has an owner.
TL;DR
After a hung transcription exposed an uncaught TimeoutExpired exception, the author split pipeline failures into infrastructure issues, which silently retry, and content issues, which log a failure event. Broken files follow a retry ladder capped at three attempts, then a visible skip that still blocks completion. Correct exit-code mapping keeps retries from amplifying hopeless failures.
# I finally split my transcription failures into two families
That night I found a long transcription process dead in the middle of a window. The process was gone, no event had been written, and one work row just sat there. My first guess: it died silently, vanished without a trace. That guess was completely wrong. [3] Python actually kills the child process on timeout and throws a TimeoutExpired exception to the caller. Nothing dies silently; what was broken was my code, which never caught that exception. I had treated every failure as the same shape, and that was the real mistake.
Two families: whoever owns the fix
After that incident I split every failure in my media pipeline into two big families, judged by who owns the repair. Family one: infrastructure failures. The [1] whisper-cli binary dies or hangs until the timeout fires, ffprobe never runs, the machine runs out of memory. That is the machine's problem, not the file's. This family maps to exit 5: no event gets written, just retry. Family two: content failures. The file itself is the problem, for example a broken container ffmpeg cannot read at all. This family maps to exit 6: the system writes a `transcribe-failed` event and bumps the attempts counter by one.
This split is not just error-catalog tidiness. Exit codes are an API for the caller. If every failure mode is not mapped to its owner before the retry logic is written, the retry loop will amplify exactly the failures that can never succeed. A corrupt file gets retried, fails again, gets retried again, forever.
A retry ladder so failure converges
The content family needs a stopping rule. The answer is a ladder of attempts: while attempts stays below 3, the system retries automatically; once it reaches 3, the file is skipped with a skip status. The important part of this skip: it is non-state. The work row still counts as an unfinished gap and still blocks closing the pipeline. A problematic file must not quietly pass as done just because its retries ran out; it has to stay visible until a human decides.
One worry was left: windows at the end of a file. [2] The ffmpeg documentation says seeking with -ss is never exact, only to the closest seek point before the requested position. It turns out a short final window is normal output, not an error: -t as an input option simply limits how much data gets read, and at the end of a file that means reading less. No special handling needed.
A timeout is an exception, not a death
The most expensive lesson from that night: a hang never happens without a sound. [3] When the timeout expires, the child process is killed and the TimeoutExpired exception is re-raised after the process has been cleaned up. The original bug was never the timeout itself; it was the missing handler. Once I caught it and mapped it to exit 5, the work row stopped hanging in limbo. My internal test suite is now fully green, and my pipeline stopped blaming the network or the machine for files that are simply broken.
Exit codes mapped correctly are cheap to build and expensive to postpone. Clear failure families saved me from a pointless retry loop, and the attempt ladder makes sure every failure ends in a documented decision, not in noise.