devops

Fleet-wide custom Docker images make digest-based upgrade playbooks obsolete; verification shifts to app endpoints, upgrade path becomes clone→build→transfer.

TL;DR: Locally built Docker images broke my Ansible upgrade playbook, since digest comparison needs a remote registry image that custom builds never have. Automation has permanently forked for these hosts. Upgrades are now manual: build locally, transfer via scp, docker load, then swap containers with about 15 seconds downtime while data mounts stay untouched.

The Docker Hub image lags behind the npm release. Build on a big machine, ship it with save, gzip, and scp, then load it on a small VPS.

TL;DR: The Docker Hub image trailed the npm release, and a 964MB VPS can't build a Next.js app without the OOM killer intervening. The fix: build locally, save and gzip the image, scp it over, and docker load it, keeping the old container renamed for instant rollback. Custom images carry different digests, so digest-checking playbooks must skip this server.

An Ansible dry run died on a harmless debug task. The skipped command tasks never filled their registered vars, and the trailing default filter never got a cha…

TL;DR: Running an Ansible playbook with --check can crash on a debug task even when the real tasks skip cleanly, because skipped tasks never register their variables. Filter chains run left to right, so last hits the undefined value before a trailing default can help. The fix is wrapping the value in default before later filters run.

Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.

TL;DR: A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.

The 10th fleet server shattered the automation illusion: SSH MaxAuthTries burned, cloud-init overrode hardening, node_exporter missing. Lesson: fleet discipline means idempotent pipeline, not manual one-offs.

TL;DR: Adding the tenth fleet server exposed SSH hardening failures from MaxAuthTries limits and cloud-init config overrides, plus a missing node_exporter. The Ansible playbook now manages drop-in SSH configs and enforces uniform monitoring rollout. True automation means the eleventh server deploys without manual intervention.

A restart gate that refuses while the tree is dirty.

TL;DR: Restarting a bind-mounted container deploys whatever sits in the working tree, so a half-finished refactor went live uncommitted. The fix is a restart script that refuses to run when the served path is dirty, using git status porcelain. The guard narrows but doesn't close the race window between check and start.

My dev script still called docker-compose in eleven places. Compose v1 has been EOL since June 2023; only a symlink kept it alive.

TL;DR: The dev script still called docker-compose, deprecated since mid-2023, and only survived thanks to a Docker Desktop compatibility symlink masking the problem. Compose v2 turned the command into a docker CLI subcommand, so it's now two words. Swapping the hyphen for a space across all call sites is a tiny commit that prevents CI breakage.

Pushes to our staging branch kept creating throwaway preview deploys. One git.deploymentEnabled block in vercel.json ended it.

TL;DR: Vercel deploys every pushed branch by default, so a repo with main and stag ends up with redundant preview deployments. An eight-line vercel.json using git.deploymentEnabled silences stag while keeping main on production. It beats the Ignored Build Step because declarative JSON can't invert exit codes or rot, just avoid overlapping glob rules.

The diff was docs-only, yet the deploy migration from Cloudflare Workers to Vercel still happened. The wiring lives in the Vercel dashboard and GitHub settings, not in the repo.

TL;DR: A docs-only commit revealed the Workers-to-Vercel migration left almost no trace in Git, since the wiring lives in Vercel and GitHub settings. Vercel elects the production branch only at import, so the GitHub default branch stag and production branch main are two independent switches. Env variables need per-target re-registration, and rollback is repointing a deployment, not git revert.