Registering KTM: When the 10th Server Broke the Automation Illusion
The 10th fleet server shattered the automation illusion: SSH MaxAuthTries burned, cloud-init overrode hardening, node_exporter missing. Lesson: fleet discipline means idempotent pipeline, not manual one-offs.
TL;DR
Adding the tenth fleet server exposed SSH hardening failures from MaxAuthTries limits and cloud-init config overrides, plus a missing node_exporter. The Ansible playbook now manages drop-in SSH configs and enforces uniform monitoring rollout. True automation means the eleventh server deploys without manual intervention.
The Moment a New IP Hit the Inventory
I still remember the exact moment when IP 157.245.192.183 appeared in hosts.yml. Commit 668f0ed in the server-old repo documents when I had to register KTM as the 10th fleet server. At first I thought, "Just run the Ansible playbook, done." I assumed the config rollout would be smooth like the previous nine servers. Turns out, that guess was completely wrong.
Instead of success, the Prometheus dashboard went red with scrape errors. SSH logs showed authentication failures piling up. The new server had no node_exporter — and worse, PasswordAuthentication was still active even though I thought I'd locked down all access.
The Lexical Order and MaxAuthTries Trap
The first problem showed up when I tried to SSH in. My ed25519 key was rejected and the connection dropped immediately. I initially thought the public key hadn't been copied yet. But the issue was the attempt limit.
OpenSSH has a MaxAuthTries mechanism defaulting to 6 attempts per connection (sshd_config(5)). Every key offered and rejected burns one attempt slot. Because my SSH agent's key order was off, the wrong key got offered first. This consumed the attempt budget before the correct key could even be verified.
I realized SSH hardening isn't just about disabling PasswordAuthentication in the main sshd_config.
The Illusion of the Main Config Being Enough
This is where my fatal mistake lay. I thought changing sshd_config was sufficient. Reality shows that cloud-init often drops a 50-cloud-init.conf file in /etc/ssh/sshd_config.d/ containing PasswordAuthentication yes.
This default config effectively overrides my main settings because of the first obtained value wins rule and Include globs processed in lexical order (sshd_config(5)). Files with higher numeric prefixes win. So my hardening attempt lost to the cloud provider's config. I had to force Ansible to manage those drop-in files explicitly.
Monitoring Gap: Nine Servers Clean, the Tenth Missing
On top of that, the new server had zero monitoring agent. Node Exporter is the standard agent for collecting hardware and OS-level metrics from Linux servers, exposing CPU, memory, disk, network, and filesystem in Prometheus format [1]. Every server you want to monitor needs Node Exporter running, making it a perfect candidate for Ansible automation.
Because previous rollouts were done manually for some nodes, this one slipped through the cracks.
Real Fleet Discipline
This incident forced a mindset shift. Adding a new node isn't just about adding one IP line. It's about ensuring the entire rollout pipeline is idempotent. I revised the Ansible playbook so monitoring and SSH hardening roles apply uniformly. Including overwriting or removing those stubborn cloud-init config files.
Going forward, I won't rely on the assumption that "the old playbook will just work." Every new server is a real test of the automation architecture I built. If the rollout still needs manual intervention or a fire drill, the system isn't mature yet. True automation shows itself when adding the 11th server tomorrow morning runs without me touching a keyboard.