The Watchdog Still Checked a Container Path That Was Gone
Migrating the Hermes watchdog to host-native was not just swapping paths. Lessons in git diff --quiet, set -euo pipefail, and a zero-token pattern.
TL;DR
Migrating Hermes to host-native broke hermes-patch-watchdog.sh, which still pointed at the old container path. The fix required more than a find-and-replace: git semantics shifted, the venv moved, and set -euo pipefail turned silent mid-script failures into false healthy signals. Every check now fails loudly, since for a silence-means-healthy watchdog, noise beats quiet mistrust.
The migration of Hermes from containers to host-native was done. Containers were shut down, every service ran directly on the host. One file slipped past my checks: hermes-patch-watchdog.sh, a small script that audits runtime health every morning. It was still faithfully pointing at /opt/hermes, the old container-era path that no longer exists. The irony: the watchdog is the script that should be most aware of environment changes, yet it was the last one I thought to move.
More Than Swapping a Path Prefix
My first guess was simple: open the script, replace /opt/hermes with the home path, done. Comparing the old and new diffs showed three behaviors that changed along with it, not just a folder address.
First, the meaning of the git check shifted. The repo at /opt/hermes used to be where patches lived, so git status --porcelain on the plugin folder directly showed whether patches had been overwritten. Now the repo is vanilla and my patches live on a separate branch. A clean repo is the normal state, not a danger sign. The new version therefore uses git diff --quiet [3], which silences all output and makes the exit status 1 when there are differences, then only raises an alarm when modifications touch the plugin or web folders.
Second, the venv moved too. The old script called python from a .venv inside the container path; the new one points to a venv inside the host-native repo. It looks trivial, but if the python path is wrong, the turn_log check fails without a clear message.
Third, set -euo pipefail is attached to the new version. And this is the part that made me think twice.
The set -euo pipefail Trap in a Zero-Token Pattern
The Bash manual explains that set -e makes the shell exit immediately when a pipeline returns a non-zero status [1], except when the command sits inside an if, an and-or list, or a few other exceptions. pipefail adds a rule: a pipeline's return value is set by the rightmost command that fails [1]. Add minus-u, which turns any unset variable into an instant error [1]. On paper, a great default for any script.
The problem is that this watchdog runs on a zero-token pattern: empty stdout means healthy, any output means alarm. The combination has one sneaky failure mode. If a command in the middle of the script dies for a small reason, say the checked file got renamed by another process, set -e can cut the script right there. The remaining checks never run, output stays empty, and other systems read that as a healthy day. A failure dressed up as success.
The fix is not going back to a guard-free script, but wrapping every check in an explicit condition. Intentional failures print an alarm; unexpected failures must never pass in silence. The principle is simple: for a script whose contract is silence-means-healthy, it is better to be noisy on error than quiet and wrongly trusted.
What Stayed: DB Size and turn_log
The core checks did not change, and there are only two. First, the size of memory_store.db plus its WAL file, with an alarm past 500MB. In WAL mode, changes are appended to the wal file first, then folded back into the database by a process called checkpointing [2]. The automatic checkpoint runs when the wal file reaches 1000 pages [2], so growth is normal, but on a small machine it still needs watching so it does not quietly eat disk.
Second, the turn_log line count. The rotation cap is 5000 and the alarm fires at 4800, right against the ceiling. Numbers like these exist in no documentation; they come purely from the runtime's own behavior, and that is exactly why this script exists.
One common assumption to correct: a script like this must run on a systemd timer, and a timer unit is indeed designed to bring a service up when its time arrives [4]. In my setup it actually runs through Hermes' internal cron on the 0 8 * * * schedule read from a local jobs.json. Which runner it uses does not matter; what matters is that the alarm still reaches the right place.
The lesson I keep: when moving house, auditing scripts that cling to absolute paths is mandatory, not a find and replace. These scripts are the only ones that truly understand the physical shape of the system, and they are the easiest to rot invisibly.