The Ansible task that failed because conf.d did not exist
The mitigation playbook died on the ninth server: journald.conf.d had never existed. One file state=directory task closes that assumption for good.
TL;DR
An Ansible playbook copying a journald cap into /etc/systemd/journald.conf.d failed on one server because that directory never existed there; the other eight just happened to have it. The fix is one directory task with state=directory placed before the copy. Lesson: every playbook assumption, like a parent directory existing, should be an explicit idempotent task.
It failed on the ninth server
My mitigate-disk.yml playbook had crossed eight servers without drama. On the ninth, BSA-WEB, it stopped halfway: the task copying the SystemMaxUse=300M cap into /etc/systemd/journald.conf.d/ failed. My first guess was a typo in the task, or a different path on that host. Digging in, the cause was sillier: the directory /etc/systemd/journald.conf.d had simply never existed on BSA-WEB. My task was copying a file into a folder that was not there.
That is where I noticed the hidden assumption inside the playbook itself. The eight other servers happened to have that directory, so the task kept passing and was never challenged. "Works on eight servers" is not proof the playbook is correct; it is proof eight servers happen to agree.
The journald drop-in is an official convention
The cap I installed is not a hack. The journald man page documents SystemMaxUse= as enforcing size limits on journal files [1], and drop-ins from /etc/systemd/*.conf.d/ carry higher precedence than the main file [1]. The systemd documentation even recommends drop-ins over editing the main file [1]. So the direction was right; what was missing is one thing: the drop-in convention does not guarantee the folder exists.
Stock installations do not always ship journald.conf.d, and that is legitimate for systemd, because default configuration is fixed at compile time [1]. The folder only appears when someone creates it. A playbook that writes into it is therefore a playbook that must create it first.
The boring one-task fix
The fix is as boring as it gets: one task before the copy task.
- name: Ensure journald.conf.d exists
when: ansible_service_mgr == 'systemd'
ansible.builtin.file:
path: /etc/systemd/journald.conf.d
state: directory
mode: "0755"
The ansible.builtin.file module manages attributes of files, directories, or symlinks [2], and with state=directory all intermediate subdirectories get created when missing [2]. On the eight other servers this task becomes a no-op, because the folder is already there; the playbook stays idempotent. On BSA-WEB it does one job, and the copy task behind it finally has somewhere to land.
The order cannot be flipped. Copy first, mkdir later just schedules the same failure again. To be blunt: the config file you copy is not the deliverable; the complete path down to that file is. A task that writes config without guaranteeing its parent exists carries a bomb whose fuse depends on host luck.
One small detail is easy to miss: this task carries when: ansible_service_mgr == 'systemd'. Non-systemd hosts have no journald at all, and forcing a journald directory onto such a host only produces config litter. The fact that this condition looks trivial is exactly the point: every playbook assumption deserves to be written explicitly, not kept in its author's head.
A fair question follows: why did the eight other servers have the folder while BSA-WEB did not? The honest answer: I do not know, and it does not matter for the fix. What matters is the pattern. Base images differ, and the drop-in convention is designed for exactly this: files under /etc/ belong to the local admin, while vendor packages put theirs under /usr/ [1]. Automation that writes into the local admin's area must be ready for hosts that are still pristine.
The pattern also travels well to other configs. Whenever a task copies into some kind of conf.d, whatever the application, a directory task in front is its mandatory partner. The cost is one no-op task on hosts that are already tidy, and one savior on hosts that are not. Compare that with the reverse cost: a playbook failing halfway on one server, lost incident hours, and a disk cap that never got installed.
The closing numbers
Once the directory was guaranteed and the cap installed, BSA-WEB dropped from /var/log 992M to 458M, and the journal from 720M to 288M — safely under the 300M cap. The logrotate rule for btmp also landed on the same host; I already wrote up its guard separately in the logrotate guard article.
Total code change for this whole outcome: seven lines of file task. That is what I like about this kind of pattern. The problem came from an assumption nobody wrote down, and the solution is just writing that assumption down as a verifiable task. The playbook did not get smarter; it stopped assuming.