Lock stolen mid-run: what fencing tokens taught me
A lock checked once at startup guards the door, not the data. Fencing tokens validated at every state write, plus a heartbeat lease, are what guard the data.
TL;DR
A force-killed transcribe process showed why one-time lock checks fail: after the lock expired, a second session took over and idempotent resume reused finished windows from sidecars. The fix is fencing tokens, checked at every state write, so stale sessions die with lock-stolen-mid-run. Minimum package: TTL lock, heartbeat, and a negative control proving the guard fails.
That night, in selftest R66: a transcribe process running three windows. Windows one and two had finished, their sidecars already sitting on disk. Then the process got force-killed mid-run. The abandoned lock slowly expired, a second session took over, and the resume didn't start from zero because finished windows were recognized from their sidecars and reused. My favorite part was actually scenario (C) of the same test: strip a component called heartbeat-renew from a copy of the kernel, then demand that the copy die in a predictable way when its lock gets stolen. My old assumption about locks, that checking once at the start was enough, was completely wrong.
Checking at the door is not enough
The comfortable old model treated the lock file as an entry ticket. A session that held it at the start felt entitled to run until completion. The problem shows up when the job runs long. etcd documents a lease as a client-liveness mechanism: the cluster grants it with a TTL, the lease expires if no keepAlive arrives within that period, and every key attached to the lease gets deleted along with it [2]. The loose translation: expiry is normal behavior, not an exception, and a second session taking over an expired lock is completely legitimate.
The uncomfortable part: the first session is not necessarily dead. On Linux, SIGKILL cannot be caught, blocked, or ignored [3], but a process can also stall for a long time without dying, and when it wakes up it has no natural way to learn that ownership has changed. End-of-process cleanup is unreliable, because nothing guarantees the process ever gets there. The only places every session must pass through are the points where it writes.
Fencing tokens: late writes get rejected
The pattern that closes this gap is the fencing token. Kleppmann's summary: every write request must carry a token, a number that increases every time the lock is acquired, and the storage side rejects any write whose token is smaller than one it has already processed [1]. When an old session wakes up and tries to write with a stale token, the rejection is not luck; the check lives on the storage side, at every write.
In the ingest kernel the practical shape is simple. One token-checking function is called at every state-write point, whether events, transcription, or media. If the token a session holds doesn't match the current lock contents, this session was stolen mid-run: the process stops immediately and writes nothing, with the reason string lock-stolen-mid-run. Meanwhile the legitimate session doesn't fear premature expiry, because every event extends the lock, much like a lease's keepAlive. Checking once at the door is cheaper, sure, but it only guards the door, not the data.
Idempotent resume, and a guard that must be able to fail
Two more lessons from this commit. First, resume must be idempotent: the same plan_id is reused, finished windows are recognized from sidecars, and a killed run writes no duplicate events. That's a requirement, not a bonus. If the second session had to redo the work from zero, the whole story above gets much more expensive.
Second, the negative control. A kernel copy was deliberately built without heartbeat-renew, then fed into the lock-theft scenario. The demanded outcome is failure: the copy must die with lock-stolen-mid-run. A guard that has never been seen failing isn't finished being tested, and a negative control is the cheapest way to watch it fail without waiting for a real incident. I've started treating this pattern as the minimum package for any job that writes to a shared ledger: a lock with a TTL, a heartbeat, and a token check at every write. The expensive part isn't the implementation; it's the first incident waiting for whoever postpones the pattern.