Node Exporter Rollout KTM Fleet and the MaxAuthTries Trap
Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.
TL;DR
A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.
TargetDown Alert and the Wrong Guess
My monitoring screen suddenly lit up red. Prometheus UI showed a TargetDown alert for IP 157.245.192.183 on port 9100. I immediately guessed this was trivial — just add a scrape target to the config, since node_exporter was surely already running on the tenth server. The guess was completely wrong. When I tried to SSH in, the connection dropped instantly. Turns out the node_exporter binary wasn't there at all. Even worse, the Ansible run to deploy it failed completely. The SSH authentication limit had been exhausted before the process even started.
Root Cause Behind the Ansible Failure
Let me break down why Ansible could fail like this. OpenSSH has a built-in defense mechanism called MaxAuthTries. This parameter limits the number of authentication attempts allowed per connection (sshd_config(5)). The default is typically 6 attempts per connection. The problem is, every SSH key that gets rejected or offered in the wrong order counts as one failure. When Ansible tried to offer multiple ed25519 keys sequentially, this limit was hit immediately. The connection was forcibly dropped. As a result, the entire task chain to download and run node_exporter as a single static binary — reliable across all server environments — never actually executed to completion.
I briefly thought this was a network or firewall issue. It turned out to be purely a configuration discipline problem. Access security must be solved first. Only then can monitoring be installed.
Idempotent Solution and Fleet Lesson
The fix I applied was radical but effective. I completely reordered the playbook execution. The first step is now SSH hardening. Authentication is key-only. Key presentation order is cleaned up so it doesn't burn the attempt budget. Once access is stable, only then do I run the Ansible role for node_exporter. This approach makes the rollout process idempotent. The eleventh server later won't need a custom script. The same playbook is all I need to run.
The TargetDown alert as explained in the official wiki is actually a valuable early warning for anyone managing server infrastructure seriously. Never ignore the infrastructure foundation order. Monitoring matters, but locked-down access is a non-negotiable prerequisite.