A reboot playbook brave enough to refuse the reboot
Rebooting one production server is easy; the hard part is the fencing. Lock pre-flight, the reboot module, and an uptime post-check that dares to fail.
TL;DR
An engineer built an Ansible playbook to reboot one server safely for kernel patching. The playbook guards against races with unattended-upgrades using lock checks, reboots, then verifies the operation by comparing uptime before and after. Manual invocation only, backed by 111 unit tests, because automated reboots turn one bug into nine downed servers.
The riddle that fools you
My collector watches nine servers, and this week one of them needed a kernel patch that only completes after a reboot. My plan was simple: build one Ansible playbook to reboot a single server safely. The hard part turned out to be not the reboot command but the fences around it. The reboot.yml playbook is mostly refusals rather than actions: it can refuse to run, and it can fail after running.
My first assumption: how hard can it be, the ansible.builtin.reboot module already handles everything. The docs are clear — it reboots the machine, waits for it to go down, come back up, and respond to commands again [1]. With reboot_timeout at 600 seconds to cover the down and up cycle [1], I could fire it and walk away.
But look at the sequence. Those nine servers get patched automatically by unattended-upgrades, which is designed to run periodically through the apt-daily-upgrade service [3]. If my playbook charges in while a patch run is active, I cut dpkg in the middle of configuring packages. The risk is not the machine dying; it comes back up with half-finished package state. And after the reboot completes, the module only answers "done" — proof that uptime actually reset is my own problem.
Fences before and after
The solution is one playbook with three layers. Layer one, the invocation gate: this playbook requires explicit confirmation and a --limit to a single host. Without that, one typo in the inventory reboots all nine servers at once.
Layer two, pre-flight. The playbook refuses to run if an unattended-upgrades progress marker exists, or if apt and dpkg locks are still held by other processes. The lock detector is fuser: it displays the PIDs of processes using a given file or file system [2], and it returns a non-zero exit code when no process uses that file [2]. That return code makes fuser a comfortable automated probe: no output parsing, just trust the exit code.
Layer three, execution and proof. The reboot task uses ansible.builtin.reboot with a 600-second timeout [1]. Once the machine responds again, the playbook runs the post-check that sometimes makes me smile: it compares uptime before and after, and fails the host if uptime did not reset. The final recap prints before-and-after uptime, kernel, and free memory, so the evidence lands neatly in the output.
Why the failing post-check matters
A step designed to fail right after its own success does look awkward. But the distinction is clean: a task that answers "ok" only says the command ran, while a reset uptime proves the event I wanted actually happened. I learned the difference reading the module's test_command option, the standard for "the machine is ready for further tasks" [1]. Readiness says nothing about the reboot itself. Post-reboot verification must measure the reboot's effect, not the machine's responsiveness.
The second layer matters just as much: when unattended-upgrades is mid-run, what I need is not a forced reboot but a playbook that backs out cleanly. The server will finish patching, and the reboot can be retried later.
Before any of those layers ever touched a real server, the playbook went through a syntax-check and 111 unit tests. That 111 is not decoration; a reboot is the kind of operation where one mistake is felt immediately, and tests are the cheap place to find a typo'd variable or an inverted condition. Only after all green does this playbook earn the right to be called.
The outermost layer is not code at all: this playbook is manual-invocation only, never wired into the collector. Some tools belong deliberately far from automation. The collector runs on its own every few hours; an auto-triggered reboot playbook means one bug becomes nine servers down at once. The distance between "a tool a human calls" and "a tool a schedule calls" is annoying to keep, but the annoyance is cheap. Data recovery is expensive.
Sources
[1] ansible.builtin.reboot module, Ansible docs