Two Silent Killers in One Ansible Deploy

At 16:47 UTC my collector cron died. The log left a single line: ansible-playbook exited with code 4, without one line of playbook output. My first guess was the network. My second guess: the SSH key was gone. Both wrong, and what I found instead was two bugs at once, neither of them making a sound.

The Apostrophe That Was Not a Comment

I started with the strangest error: the playbook never ran at all. The trigger turned out to be a comment inside a shell task written as a block scalar, and the comment contained the word today's, apostrophe included.

- name: ssh auth probe
  ansible.builtin.shell: |
    LOG=/var/log/auth.log
    # both units unknown -> re-check today's file lines
    f24=$(journalctl -u ssh --since -24h | grep -c Failed)

What I had never realized: inside a block scalar, that # line is not a YAML comment. YAML sees the whole block as a free string, then Ansible runs it through its argument splitter. A single apostrophe makes the splitter treat the quote as unbalanced, and Ansible throws the message "failed at splitting arguments, either an unbalanced jinja2 block or quotes" [4]. One character, and the playbook is completely dead. The error even points at the task line, not at the YAML file itself.

The bonus confusion was exit code 4. I assumed it was a dedicated parser-error code. In the ansible-core source, HOST_UNREACHABLE = 4 and PARSER_ERROR = 4 share the same number, and a developer comment in the code admits the clash [3]. So exit code 4 is not a diagnosis, just a direction.

Keys Rejected Before Their Turn

After cleaning the comment, the playbook ran, and the host stayed unreachable. The cause lived outside the playbook: the collector container ran ssh without pinned identities, and ssh offered id_ed25519 first. The server refused it. Every refusal burns an authentication attempt, and MaxAuthTries defaults to just 6 per connection [5]. Before the RSA key that actually worked got its turn, the connection was dropped with "Too many authentication failures".

The fix lives in ansible.cfg: IdentitiesOnly=yes so ssh offers only the explicitly configured identities [2], plus an ordered list of IdentityFile entries mirroring the order in the host's ~/.ssh/config.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -o IdentitiesOnly=yes -o IdentityFile=/root/.ssh/id_rsa1 -o IdentityFile=/root/.ssh/id_rsa2 -o IdentityFile=/root/.ssh/id_rsa3 -o IdentityFile=/root/.ssh/id_rsa4

One trap nearly got me again: overriding ssh_args replaces the entire string, and the default contains -C -o ControlMaster=auto -o ControlPersist=60s [1]. So I rewrote the multiplexing options too, otherwise the playbook crawls from opening and closing connections over and over.

What I Changed So It Stays Fixed

Two layers of prevention. First, for shell and command tasks I moved to the dictionary form (cmd: plus args:); ansible-lint ships a no-free-form rule precisely because free-form "can produce subtle bugs" [6]. Second, comments inside long script blocks now get written without apostrophes, or the script moves to its own file. The ssh client also applies first-value-wins per directive [2], so the order of lines in a config is not cosmetics, it is semantics.

Details as small as a comment or a key order decide whether the deploy succeeds or whether I stay up all night. Five minutes checking the config is cheaper than an hour of debugging in production. Now whenever a playbook of mine dies with exit code 4, I check those two things before suspecting the network.

Sources

[1] Ansible ssh connection plugin docs

[2] ssh_config(5), OpenBSD

[3] ansible-core ExitCode constants

[4] ansible-core split_args source

[5] sshd_config(5), OpenBSD

[6] ansible-lint no-free-form rule