devops

Pushes to our staging branch kept creating throwaway preview deploys. One git.deploymentEnabled block in vercel.json ended it.

TL;DR: Vercel deploys every pushed branch by default, so a repo with main and stag ends up with redundant preview deployments. An eight-line vercel.json using git.deploymentEnabled silences stag while keeping main on production. It beats the Ignored Build Step because declarative JSON can't invert exit codes or rot, just avoid overlapping glob rules.

The diff was docs-only, yet the deploy migration from Cloudflare Workers to Vercel still happened. The wiring lives in the Vercel dashboard and GitHub settings, not in the repo.

TL;DR: A docs-only commit revealed the Workers-to-Vercel migration left almost no trace in Git, since the wiring lives in Vercel and GitHub settings. Vercel elects the production branch only at import, so the GitHub default branch stag and production branch main are two independent switches. Env variables need per-target re-registration, and rollback is repointing a deployment, not git revert.

The mitigation playbook died on the ninth server: journald.conf.d had never existed. One file state=directory task closes that assumption for good.

TL;DR: An Ansible playbook copying a journald cap into /etc/systemd/journald.conf.d failed on one server because that directory never existed there; the other eight just happened to have it. The fix is one directory task with state=directory placed before the copy. Lesson: every playbook assumption, like a parent directory existing, should be an explicit idempotent task.

Rebooting one production server is easy; the hard part is the fencing. Lock pre-flight, the reboot module, and an uptime post-check that dares to fail.

TL;DR: An engineer built an Ansible playbook to reboot one server safely for kernel patching. The playbook guards against races with unattended-upgrades using lock checks, reboots, then verifies the operation by comparing uptime before and after. Manual invocation only, backed by 111 unit tests, because automated reboots turn one bug into nine downed servers.

Internet-facing servers pile up failed-login logs with no guard. One logrotate drop-in, the su directive, and a logrotate -d gate close the problem for good.

TL;DR: btmp logs failed SSH logins and can grow past hundreds of megabytes between weekly rotations. A drop-in logrotate rule using maxsize 50M rotates oversized files early, while rotate 2 and compress keep archives small. Validate with logrotate -d before deploying; add su root utmp or logrotate refuses on group-writable directories.

A weekly playbook said OK while /var/log grew to 654M: rotated btmp archives, an active file to shrink, and a df pipe that never ran.

TL;DR: The dashboard caught disk creep my weekly playbook kept missing: years of SSH brute-force attempts had quietly stuffed btmp past 150M. Worse, the evidence task ran df through the command module, which ignores pipes, so it never filtered output. The fix swaps in shell and truncates oversized logs instead of deleting them, keeping sshd writing happily.

Two numbers for one server: risk score and health score point in opposite directions, and my sidebar was still reading the old one.

TL;DR: A dashboard showed conflicting scores for the same server because the sidebar still read the old risk-based fields (100 means critical) while everything else used the new health-based ones (100 means healthy). The lesson: invert metric direction once at the API boundary, never per component. Treat opposite-direction metric renames as data contract migrations, not simple renames.

The hardening playbook said changed, but sshd -T still showed passwordauthentication yes: first-match-wins and lexical Include order let 50-cloud-init.conf win.

TL;DR: OpenSSH uses first-match-wins for sshd config, so a drop-in named 50-hardening.conf lost to cloud-init's 50-cloud-init.conf alphabetically. Fix: rename it 00-hardening.conf, comment out conflicting lines in the main config, and edit 50-cloud-init.conf directly. Always verify with sshd -T instead of trusting a playbook's green status.

An Ansible playbook died over an apostrophe inside a block scalar, then hosts went unreachable over ssh key offer order. Two lessons from one exit code 4.

TL;DR: A comment with an apostrophe inside a shell block scalar killed the whole playbook, since YAML ignores # in block scalars and Ansible's argument splitter chokes on unbalanced quotes. Exit code 4 is ambiguous anyway. Then MaxAuthTries exhausted wrong-key attempts; IdentitiesOnly=yes with ordered IdentityFile entries fixed it.