My Playbook Said Changed, sshd Said Still Open

An SSH hardening playbook had just finished on one of my hosts, everything green, the last task reporting changed. Then I ran sshd -T | grep -i password to confirm, and the screen still showed passwordauthentication yes. My first guess: the drop-in file never got written to disk. Completely wrong. The file was written; what was wrong was my assumption about how sshd reads its config.

First Match Wins, the Rest Is Decoration

OpenSSH has a rule that is easy to forget: unless noted otherwise, for every keyword, the first obtained value is the one used [1]. Not the last one, not the most specific file. Once sshd finds PasswordAuthentication yes earlier in its reading order, every later no is decoration.

And "earlier" is sneakier than it sounds. The Include directive in sshd_config expands glob patterns and processes them in lexical order [1]. cloud-init ships a file called 50-cloud-init.conf containing PasswordAuthentication yes. My drop-in was named 50-hardening.conf. Both load from the Include, and alphabetical order decides: c before h. There is no "last file overrides" spirit anywhere, pure first-match-wins.

I am not alone. In cloud-init issue #5934, an Ubuntu 24.04 user found that 50-cloud-init.conf with PasswordAuthentication yes overrode the no in their main config, and they only discovered the file after grepping PasswordAuthentication across /etc/ssh. The issue was closed as "not planned" [3]. Same symptom, same cause.

Three Steps That Finally Worked

First, rename the drop-in to 00-hardening.conf so it beats every 50-* file lexically:

# /etc/ssh/sshd_config.d/00-hardening.conf
PasswordAuthentication no
PermitRootLogin prohibit-password

A drop-in alone is still not enough. If the main sshd_config has active PermitRootLogin or PasswordAuthentication lines sitting above the Include, those lines still beat every drop-in. My playbook now comments out such lines with the lineinfile module and its backrefs option [4], and rewrites PasswordAuthentication inside 50-cloud-init.conf itself to no. Before it runs, a sshd_config.bak- snapshot is taken, and sshd is reloaded, not restarted, so live sessions survive.

One detail kept tripping my verification: sshd -T prints permitrootlogin without-password, while the playbook writes prohibit-password. Both spellings mean the same thing, root login with keys only, so the idempotency and verification checks accept both. Without that, the playbook would consider the host "not hardened" forever and rewrite the config on every run.

Verify the Effective Config, Not the File You Wrote

The playbook stays manual-only: it requires -e confirm=yes, demands --limit per host, and is never wired into cron or automation. Per my repo policy, state-changing operations need explicit human approval per host. The irony is that precisely because a playbook like this "only runs once", the temptation to trust its green status grows.

The lesson I took away: verify the effective configuration, not the file you wrote. sshd -T prints the final result of every layer after first-match rules and Include ordering have had their say. A carefully written file means nothing if the daemon reads it with precedence rules you ignored. Every playbook of mine that touches sshd now ends with an sshd -T check instead of a changed=1 recap. Small thing to write down, big difference when a real incident hits.

Sources

[1] sshd_config(5), OpenBSD

[2] ssh_config(5), OpenBSD

[3] cloud-init issue #5934: Cannot disable PasswordAuthentication due to 50-cloud-init.conf

[4] Ansible lineinfile module docs