When a version monitor wakes you for an intentional state

Five in the morning, my terminal showed a single red line from 9router-version-watch.py: PROBLEM, fleet out of sync with Docker Hub. My first instinct was the wrong one: some host must have failed its upgrade, just find which one. I opened the per-host version list one by one, ready to roll things back. Reality was embarrassing in the best way: all six hosts were healthy, uniform, and exactly where I had pushed them the day before, the custom 0.5.81 build. The thing left behind was not the fleet but Docker Hub itself, still sitting at 0.5.75 [6]. The monitor woke me up for a condition I had designed on purpose.

That was when I realized my initial guess had been wrong at the foundation. I was hunting for a broken host while the broken part was the comparison logic. The script had assumed from day one that Hub latest was the single source of truth: fleet not equal to Hub means something is wrong. But a registry can lag behind an approved reality, and without a concept of direction, "different" automatically reads as "broken".

Version comparison has a direction

The fix was only about seventeen lines, but it forced me to rebuild how I think about comparing versions. First, version order must never be compared as strings. 0.5.9 looks "greater than" 0.5.75 lexicographically, even though as a version it is clearly smaller. The SemVer spec defines MAJOR.MINOR.PATCH to be compared per numeric component: MAJOR for breaking changes, MINOR for compatible new features, PATCH for bug fixes [1]. So I added a small helper called _vkey: it splits the version string into a numeric tuple, so 0.5.81 really is greater than 0.5.75 the way a human expects.

Second, and this is the part that matters most: the direction of the difference has to carry its own meaning. The new gate 2b states: if the entire fleet is uniform and newer than Hub, that is an approved custom rollout. The script stays silent with exit 0 and keeps watching for Hub to catch up. Mixed versions across hosts, or a fleet older than Hub, still fire PROBLEM exactly as before, because those two conditions are genuinely dangerous. The reassuring part: this pattern is not something I invented. Kubernetes' official version skew policy is explicit about direction: kubelet may be up to three minor versions older than kube-apiserver, but must never be newer [4]. Direction of skew always carries a different operational meaning.

The rest of the world already has words for this

After the incident I went back to reading docs, and it turns out the "looks wrong but is intentional" state has long been recognized across different ecosystems. npm, for example: publishing a package moves the latest tag automatically, unless you deliberately publish with a different --tag like beta or canary [5]. Which means what runs in production can be far newer than what a default install sees, and that is documented, normal behavior. If I had built a monitor comparing "what runs in production" against "what npm install sees", I would get precisely the same false alarm.

What annoyed me most was not the bug, it was the insomnia I had engineered for myself. Healthy alerting is well understood: the SRE Workbook puts it as the goal of being notified only for a significant event, one that actually consumes a large fraction of the error budget [2]. The Prometheus alerting practices summarize the same philosophy more bluntly: keep alerting simple, alert on symptoms, and avoid pages where there is nothing to do [3]. A five a.m. PROBLEM for a healthy fleet is a textbook page with nothing to do. Six hosts, zero actions, one human losing sleep.

The lesson I took home: a monitor is only as good as its understanding of "intentional" states. As long as I had not taught the script about approved custom rollouts, it would keep reporting my own decisions as failures. Seventeen lines of _vkey and gate 2b bought something far more valuable than a quieter log: I now know exactly which conditions deserve to wake me, and which ones just get recorded and left alone.