ssh

db-push replaced staging from a dev dump and accounts vanished. Two small dumps and honest guards keep them alive.

TL;DR: Staging users kept vanishing because db-push replaces the whole database with a dev dump that excludes the users table. The fix: dump staging users beforehand, then restore them after the import using --no-data, --complete-insert, and --no-create-info. Key lesson: dump and verify before any DROP, validate injected names, and lock down the hash-bearing file.

Flipping db-push into db-pull is more than reversing arrows: local backs up first, the dump gets inspected before trusted, backtick rules change over SSH.

TL;DR: Reversing the sync direction puts your local database at risk, so the guard sequence has to flip. The script first backs up local data for rollback, then takes a consistent staging dump and checks size, tables and completion before wiping anything. It avoids SSH backtick issues and strictly compares row counts afterward, preferring a hard failure over a half-finished import.

The Docker Hub image lags behind the npm release. Build on a big machine, ship it with save, gzip, and scp, then load it on a small VPS.

TL;DR: The Docker Hub image trailed the npm release, and a 964MB VPS can't build a Next.js app without the OOM killer intervening. The fix: build locally, save and gzip the image, scp it over, and docker load it, keeping the old container renamed for instant rollback. Custom images carry different digests, so digest-checking playbooks must skip this server.

Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.

TL;DR: A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.

The 10th fleet server shattered the automation illusion: SSH MaxAuthTries burned, cloud-init overrode hardening, node_exporter missing. Lesson: fleet discipline means idempotent pipeline, not manual one-offs.

TL;DR: Adding the tenth fleet server exposed SSH hardening failures from MaxAuthTries limits and cloud-init config overrides, plus a missing node_exporter. The Ansible playbook now manages drop-in SSH configs and enforces uniform monitoring rollout. True automation means the eleventh server deploys without manual intervention.

The hardening playbook said changed, but sshd -T still showed passwordauthentication yes: first-match-wins and lexical Include order let 50-cloud-init.conf win.

TL;DR: OpenSSH uses first-match-wins for sshd config, so a drop-in named 50-hardening.conf lost to cloud-init's 50-cloud-init.conf alphabetically. Fix: rename it 00-hardening.conf, comment out conflicting lines in the main config, and edit 50-cloud-init.conf directly. Always verify with sshd -T instead of trusting a playbook's green status.

An Ansible playbook died over an apostrophe inside a block scalar, then hosts went unreachable over ssh key offer order. Two lessons from one exit code 4.

TL;DR: A comment with an apostrophe inside a shell block scalar killed the whole playbook, since YAML ignores # in block scalars and Ansible's argument splitter chokes on unbalanced quotes. Exit code 4 is ambiguous anyway. Then MaxAuthTries exhausted wrong-key attempts; IdentitiesOnly=yes with ordered IdentityFile entries fixed it.

One Compose service driven by FORGEJO__ env vars: built-in loopback SSH, push-to-create, LFS, and a healthcheck that tells the truth.

Every SSH login prints setlocale: LC_CTYPE cannot change locale. The sender is your own Debian laptop via its SendEnv default.