prometheus

Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.

TL;DR: A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.

The 10th fleet server shattered the automation illusion: SSH MaxAuthTries burned, cloud-init overrode hardening, node_exporter missing. Lesson: fleet discipline means idempotent pipeline, not manual one-offs.

TL;DR: Adding the tenth fleet server exposed SSH hardening failures from MaxAuthTries limits and cloud-init config overrides, plus a missing node_exporter. The Ansible playbook now manages drop-in SSH configs and enforces uniform monitoring rollout. True automation means the eleventh server deploys without manual intervention.

A noisy alert trains you to mute the channel. Debouncing 3 samples, recovering at 75% of warn, and deduping per incident before Discord pings you again.

TL;DR: A transient backup spike fired the alerts until the channel got muted, which is worse than no monitoring. The fix: three-sample debouncing, a 0.75 hysteresis factor for recovery, and per-incident dedupe, mirroring Prometheus's for and keep_firing_for clauses. Core principle: alerts must be urgent and actionable, or they're spam.

A last value without its age is a lie. Adding staleness badges (fresh/aging/dead), window-following aggregates, and portable systemd timestamps.

TL;DR: A dashboard showed 3% CPU while the service had been dead for seven hours, because the last sample was rendered as if fresh. The fix classifies data as fresh, aging, or dead based on age versus poll interval. Lesson: a value without its age is a lie.