devops

A one-byte frame tamper taught me how content-addressed sidecars, an append-only event log, and disk-truth verification keep a pipeline recoverable.

TL;DR: When a corrupted frame triggered media-quarantine instead of a crash, the kernel proved it rebuilds state from events and verifies files from disk, never trusting recorded hashes. Content-addressed caption sidecars give free deduplication, while the append-only log enables clean recovery through full rebuilds. But hash chains prove tamper-evidence, not authenticity—integrity is an architecture, not a checksum ritual.

Fleet-wide custom Docker images make digest-based upgrade playbooks obsolete; verification shifts to app endpoints, upgrade path becomes clone→build→transfer.

TL;DR: Locally built Docker images broke my Ansible upgrade playbook, since digest comparison needs a remote registry image that custom builds never have. Automation has permanently forked for these hosts. Upgrades are now manual: build locally, transfer via scp, docker load, then swap containers with about 15 seconds downtime while data mounts stay untouched.

The Docker Hub image lags behind the npm release. Build on a big machine, ship it with save, gzip, and scp, then load it on a small VPS.

TL;DR: The Docker Hub image trailed the npm release, and a 964MB VPS can't build a Next.js app without the OOM killer intervening. The fix: build locally, save and gzip the image, scp it over, and docker load it, keeping the old container renamed for instant rollback. Custom images carry different digests, so digest-checking playbooks must skip this server.

An Ansible dry run died on a harmless debug task. The skipped command tasks never filled their registered vars, and the trailing default filter never got a cha…

TL;DR: Running an Ansible playbook with --check can crash on a debug task even when the real tasks skip cleanly, because skipped tasks never register their variables. Filter chains run left to right, so last hits the undefined value before a trailing default can help. The fix is wrapping the value in default before later filters run.

Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.

TL;DR: A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.

My dev script still called docker-compose in eleven places. Compose v1 has been EOL since June 2023; only a symlink kept it alive.

TL;DR: The dev script still called docker-compose, deprecated since mid-2023, and only survived thanks to a Docker Desktop compatibility symlink masking the problem. Compose v2 turned the command into a docker CLI subcommand, so it's now two words. Swapping the hyphen for a space across all call sites is a tiny commit that prevents CI breakage.

Pushes to our staging branch kept creating throwaway preview deploys. One git.deploymentEnabled block in vercel.json ended it.

TL;DR: Vercel deploys every pushed branch by default, so a repo with main and stag ends up with redundant preview deployments. An eight-line vercel.json using git.deploymentEnabled silences stag while keeping main on production. It beats the Ignored Build Step because declarative JSON can't invert exit codes or rot, just avoid overlapping glob rules.

An Ansible playbook died over an apostrophe inside a block scalar, then hosts went unreachable over ssh key offer order. Two lessons from one exit code 4.

TL;DR: A comment with an apostrophe inside a shell block scalar killed the whole playbook, since YAML ignores # in block scalars and Ansible's argument splitter chokes on unbalanced quotes. Exit code 4 is ambiguous anyway. Then MaxAuthTries exhausted wrong-key attempts; IdentitiesOnly=yes with ordered IdentityFile entries fixed it.

A server monitoring dashboard becomes noise when attention items pile up with no organization. Three complementary fixes transform a flat grid into a usable workflow.

TL;DR: Dashboard Guy Eighteen flat grid of alerts made it impossible to find critical issues across nine servers. Backend search with debounce, severity filter buttons, and collapsible server groups solved the noise problem by pushing all filtering logic server-side. Three tools together let you triage from hundreds of items in two taps or three words.