My Phone Buzzed at 2 AM and There Was Nothing To Do

The phone buzzed at 2 AM. Discord notification: CPU on the staging server had crossed 80%. I dragged myself out of bed, opened the laptop, and found the truth: a momentary spike from the nightly backup. Five minutes later it was back under the line. Nothing to do. The more dangerous part happened by morning: the alerts had quietly turned into e-commerce promo notifications. Glance, swipe, forget.

My first guess was that nothing was wrong with the setup. Alerts should fire the instant a threshold is touched, that is what thresholds are for, and faster knowledge is better knowledge. That thinking killed the alerts socially. They were not broken, just noisy, until the channel got muted. And a muted alert is more dangerous than no alert at all: you feel monitored while you are not.

Debounce and Hysteresis, At Home

The fix lives in alerts.py, now 351 lines. Three decisions.

First, debounce before firing. Three samples, DEBOUNCE_N = 3. should_fire(avg3, votes, warn) is true only when the 3-sample average reaches warn, or when a 2-of-3 majority sits above it. A lone backup spike cannot fire anything anymore.

Second, hysteresis before recover. Opening at warn 80 and recovering at the same 80 is a broken light switch, flapping on-off-on. So recover factor 0.75: should_recover(avg3, rec) is true only when avg3 < rec, meaning warn 80 recovers below 60. That 0.75 is a local choice, not an industry standard. Prometheus attacks the same problem time-based with keep_firing_for instead of a ratio [4].

Third, incident dedupe. An in-memory dict _STATE keyed by server:kind. One breach means one alert, plus a short recovery note when it heals.

Prometheus had this exact problem and solved it with two clauses: for, which holds an alert in pending before it fires (my debounce, effectively), and keep_firing_for, which keeps firing after the condition was last met, preventing flapping and false resolutions [4]. So I am not inventing a weird pattern here. Just re-implementing a simple version at small scale.

Alerts Must Be Actionable

Google SRE has a rule every page must pass: does the rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible? If not, it is alert spam [5]. A 30-second CPU spike from a backup fails every criterion.

Recovery is a first-class event, same spirit as Nagios freshness checks treating recovery as a legitimate state [3]. One row per incident in alert_events, and recovery UPDATEs that row instead of inserting a new one.

Routing is boring on purpose. ssh_bruteforce goes to security, a container: prefix goes to availability, everything else defaults to performance.

Discord embeds have hard limits, so the formatter knows them too: MAX_FIELDS = 25 and 1024-char field values, truncated before Discord gets the chance to reject. And one fail-open decision: Discord down or the store failing means log and drop. The collector loop never gets blocked by its own alerting.

My Position Now

A non-actionable alert is not an alert. It is spam that trains you to ignore your own systems, which is worse than blindness because it feels safe.

So there is a new rule before I add any alert: if it fires at 2 AM, what will I actually do? If the answer is look, then close, it goes on the dashboard, not the phone.

The numbers are not sacred. Three samples, 0.75, server:kind, all of it negotiable. The principle is not: alerts carry the burden of proof before they may interrupt a human.

Sources

1. Alerting rules — Prometheus Documentation. "The optional for clause causes Prometheus to wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element." and "There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met." Accessed 2026-09-16.
2. Monitoring Distributed Systems — Google SRE Book. "Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?" and "Every page should be actionable." Accessed 2026-09-16.
3. Host and Service Freshness Checks — Nagios Core Documentation. "If the age of the last check result is greater than the freshness threshold, the check result is considered stale." Accessed 2026-09-16.