When Custom Docker Images Break Your Upgrade Playbook
Fleet-wide custom Docker images make digest-based upgrade playbooks obsolete; verification shifts to app endpoints, upgrade path becomes clone→build→transfer.
TL;DR
Locally built Docker images broke my Ansible upgrade playbook, since digest comparison needs a remote registry image that custom builds never have. Automation has permanently forked for these hosts. Upgrades are now manual: build locally, transfer via scp, docker load, then swap containers with about 15 seconds downtime while data mounts stay untouched.
The Registry Sync Illusion
I just finished rolling out the 5th production server. The Ansible output was still on my terminal when I opened memory-facts.md to document the change. That's when I noticed the old playbook comment "MUST skip Coreg" now applied to 7 hosts total, including the fresh KTM install. I initially assumed this custom image thing was a temporary patch — once Docker Hub caught up with the npm registry's 0.5.81 release, we'd go back to normal automation. Turns out I was completely wrong. The automation path had permanently forked.
The playbook upgrade-9router.yml had always relied on digest comparison between local and remote images to trigger updates. That logic collapsed the moment these hosts started running 9router-custom:0.5.81. This image was built locally with specific modifications, like removing registry.npmmirror.com from the Dockerfile. Docker Hub will never publish a digest for this variant because it's my own local build artifact.
That's where my fundamental mistake was — registry-based automation assumes all images come from a centralized registry. When I patched the Dockerfile and built it myself, I was effectively maintaining a fork. There's no remote digest to compare against. Realizing this made me understand that keeping the old playbook for these hosts was pointless. The fact that custom images break the playbook isn't a bug — the underlying assumption had become obsolete.
The New Upgrade Reality
The consequence: verification and deployment flows had to change completely. I could no longer rely on digest comparison. Version verification now depends entirely on the application's own endpoint at /api/version.
The upgrade process became a structured manual sequence. I have to clone the latest source tag, build locally, then save it as a tar.gz archive around 207 MB. This file gets transferred between hosts using scp, which runs SFTP over an SSH connection. On the target server, I run docker load to restore the image and its tags from the compressed archive.
Container swaps happen in-place with downtime of only about 15 seconds per host. The unless-stopped restart policy stays intact so containers survive daemon restarts. Most importantly, the data bind mount remains completely untouched. Every custom image is a fork of the automation path — and I had to redesign the workflow instead of forcing old tools onto new reality.