Dashboard Numbers Go Stale Without Telling You

Last week I opened the monitoring dashboard for a server running quietly in the background. CPU: 3%. Everything looked calm, healthy, idle.

Then I SSH'd in. The service had been inactive (dead) since 4 AM. It was 11. Seven hours completely dead, and the dashboard kept showing 3% like nothing happened.

The dashboard was not lying, not exactly. The 3% was true at 4 AM, when the collector last shipped data. After that: nothing. No updates, no errors, no red flags. Just a number sitting there, aging quietly, while the dashboard kept rendering it as if it were fresh.

Three Buckets: Fresh, Aging, Dead

This is not a new problem. Prometheus solves it at the query layer: once the most recent sample in a series is older than the lookback period, the series simply stops being returned [1]. Nagios Core has had freshness checks forever: when the age of the last check result exceeds the freshness threshold, the result is considered stale and Nagios forces an active re-check [3]. Two different systems, one idea: the age of the data matters more than the value of the data.

My version is just three buckets based on data age compared to the 180-second poll interval:

def staleness_status(checked_at, now, poll_interval=180):
    """fresh/aging/dead based on data age vs poll interval."""
    if not checked_at:
        return "dead"
    age = (now - _parse_ts(checked_at)).total_seconds()
    if age <= 2 * poll_interval:
        return "fresh"
    if age <= 5 * poll_interval:
        return "aging"
    return "dead"

The 2x and 5x multipliers are not universal constants, they are my local choices: fresh up to 6 minutes, aging up to 15, dead beyond that. The concept is what matters: data has an age, and that age must be compared against how often the data is supposed to arrive. Nagios explicitly recommends setting the threshold by hand rather than letting it be derived from the monitoring interval [3].

On the frontend, the bucket becomes a small badge in the server hero, complete with the data age in minutes. When the badge says fresh, I do not think about it. When it says aging or dead, I immediately know the numbers cannot be trusted, without opening a terminal and comparing clocks by hand.

Average, Min, Max That Follow the Tab

The second problem is from the same family: numbers disconnected from the context on screen. The metric cards used to show a delta that was always computed over a hardcoded 24-hour window, no matter which range tab was selected. The label said 24 hours while the screen showed a different range. Contradictory.

Now the volatile cards (cpu_used_pct, disk_read_kbps, disk_written_kbps, net_rx_kbps, net_tx_kbps) show avg/min/max over the window being viewed, and the delta label follows the selected tab. Pick the 1-hour tab, the delta compares against the previous hour. Pick 7 days, it compares against the previous week.

Average alone can mislead for spiky metrics. A peak of 800 Mbps against an average of 120 Mbps: two very different stories, and without min/max only one of them is visible. For network throughput, min/max is not decoration. It is the number I actually look for when someone complains the connection is slow.

WIB Timestamps That Do Not Travel

One more trap from the same commit: parsing systemd timestamps. systemd displays timestamps with local zone abbreviations, for example Fri 2012-11-23 23:02:15 CET, exactly as the documentation states [2]. Parsing one on a machine in the same zone works fine, because the parser defaults to the local timezone. But systemd itself warns: timestamps from remote systems with a non-matching timezone are usually not parsable locally, as the timezone component is not understood, unless it happens to be UTC [2]. WIB is not on the standard parser's list of known zones.

The fix is explicit handling: recognize the local zone abbreviation, map it to an offset, convert to UTC before storing. Not elegant, but the only way that does not explode when the collector and the viewer run on machines in different zones. The general lesson: never parse zoned timestamps as raw strings. Store and compare in UTC, display in the reader's zone.

A last value without its age is a lie. Dashboard numbers must appear together with their age, how many minutes since the data was collected. Without that, 3% CPU can mean a healthy server, or one that has been dead for seven hours while I stare at it.

Sources

1. Querying basics — Prometheus Documentation. "Time series are only returned if their most recent sample is less than the lookback period ago." Accessed 2026-09-16.
2. systemd.time(7) — Linux man-pages. "The timezone defaults to the current timezone if not specified explicitly." and "timestamps displayed by remote systems with a non-matching timezone are usually not parsable locally". Accessed 2026-09-16.
3. Host and Service Freshness Checks — Nagios Core Documentation. "If the age of the last check result is greater than the freshness threshold, the check result is considered stale." Accessed 2026-09-16.