linux

A 14G backup folder filled the root disk. The move-to-block-storage-with-symlink pattern relieves disk pressure without deleting backup data.

TL;DR: When the root disk hit 95 percent, the culprit was a 14GB WordPress plugin backup folder. Instead of deleting it, the fix moved the folder intact to block storage and left a symlink at the old path, dropping usage to 44 percent. Verify file counts, use absolute paths, and persist the mount in fstab to avoid dangling links.

set -euo pipefail will not save a script from swapped arguments. Whitelist guards and custom exit codes at the entry point stop the waste.

TL;DR: The script ran to completion on swapped arguments because set -euo pipefail only catches unset variables, not wrong values. The fix: eight guard lines at the top that validate the recipe path, integer, and budget using simple case globs with custom exit codes 45 and 46. Validate at the trust boundary, quote variables, keep it lightweight.

A lock checked once at startup guards the door, not the data. Fencing tokens validated at every state write, plus a heartbeat lease, are what guard the data.

TL;DR: A force-killed transcribe process showed why one-time lock checks fail: after the lock expired, a second session took over and idempotent resume reused finished windows from sidecars. The fix is fencing tokens, checked at every state write, so stale sessions die with lock-stolen-mid-run. Minimum package: TTL lock, heartbeat, and a negative control proving the guard fails.

Rebooting one production server is easy; the hard part is the fencing. Lock pre-flight, the reboot module, and an uptime post-check that dares to fail.

TL;DR: An engineer built an Ansible playbook to reboot one server safely for kernel patching. The playbook guards against races with unattended-upgrades using lock checks, reboots, then verifies the operation by comparing uptime before and after. Manual invocation only, backed by 111 unit tests, because automated reboots turn one bug into nine downed servers.

Internet-facing servers pile up failed-login logs with no guard. One logrotate drop-in, the su directive, and a logrotate -d gate close the problem for good.

TL;DR: btmp logs failed SSH logins and can grow past hundreds of megabytes between weekly rotations. A drop-in logrotate rule using maxsize 50M rotates oversized files early, while rotate 2 and compress keep archives small. Validate with logrotate -d before deploying; add su root utmp or logrotate refuses on group-writable directories.

A weekly playbook said OK while /var/log grew to 654M: rotated btmp archives, an active file to shrink, and a df pipe that never ran.

TL;DR: The dashboard caught disk creep my weekly playbook kept missing: years of SSH brute-force attempts had quietly stuffed btmp past 150M. Worse, the evidence task ran df through the command module, which ignores pipes, so it never filtered output. The fix swaps in shell and truncates oversized logs instead of deleting them, keeping sshd writing happily.