The Advisory Boar
I've written about the various
in Ansible 2, including a rewrite of the connection plugin.
Unfortunately, the problem that originally motivated the rewrite
currently remains unsolved.
If you ssh to a host for which your known_hosts file has no entry, you
are shown the host's key fingerprint and are prompted with
sure you want to continue connecting (yes/no)?. If you run ansible
against multiple unknown hosts, however, the host key prompts will just
The authenticity of host 'magpie (a.b.c.d)' can't be established.
ECDSA key fingerprint is 2a:5a:4c:4b:e0:40:de:8b:9b:e6:0f:90:45:68:89:fc.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'hobby (e.f.g.h)' can't be established.
RSA key fingerprint is 61:84:90:47:f7:0f:7b:a2:d5:09:98:6f:bb:3c:50:d9.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'raven (i.j.k.l)' can't be established.
RSA key fingerprint is ab:97:c2:7d:b6:8e:c3:ab:78:a2:20:04:af:9c:6f:2b.
Are you sure you want to continue connecting (yes/no)?
The processes compete for input, so typing “yes” may or may not work:
Please type 'yes' or 'no': yes
Please type 'yes' or 'no': yes
Please type 'yes' or 'no':
Worse still, if some of the targeted hosts are known, output from their
tasks may cause the prompts to scroll off the screen, and ansible will
appear to hang.
The solution is to acquire a lock before executing ssh and releasing it
once the host key prompt (if any) is negotiated. Ansible 2 had some code
copied from 1.9 to implement this, but it was agonisingly broken. It
wouldn't have always acquired the lock or released it correctly, but the
actual locking was commented out anyway because of lower-level changes,
so it just scanned known_hosts twice for every connection. Even if the
locking had worked, the lock would have been held until ssh exited.
I submitted patches
to add a connection locking infrastructure and use it to hold a lock
only until ssh had verified the host key (not until it finished).
Although most of the changes were merged, the actual ssh locking was
rejected because it would (unavoidably)
wait for ssh to timeout
while trying to connect to unreachable hosts.
One of the maintainers recently
said they may reconsider
this (because it's painful to deal with any number of newly provisioned
hosts otherwise), so I have
opened a new PR,
but it has not yet been merged.
Update: The maintainers went with
a different approach
to solve the problem. Instead of using locking inside the connection
plugin, this checks the host key as a separate step at the strategy
level, at the expense of having to parse the known_hosts file to check
if a host's key is already known. I think that's a fragile solution, but
it does eliminate the locking concerns and improve upon the status quo.
Another update: The commit referenced above was reverted later
the same day, for some reason the maintainers did not see fit to record
in the commit message. So we're right back to the broken starting point.
While writing about
earlier, it occurred to me that pipelining could be made to work with
requiretty, thus saving having to edit /etc/sudoers,
and even making it possible to use su (which always requires a tty).
This would mean pipelining could be enabled by default, for a noticeable
for gory details) that I've submitted as a PR for Ansible 2. Let's hope
it's merged soon.
is an Ansible feature to reduce the number of connections to a host.
Ansible will normally create a temporary directory under
~/.ansible (via ssh), then for each task, copy the module
source to the directory (using sftp or scp) and execute the module (ssh
With pipelining enabled, Ansible will connect only once per task using
ssh to execute python, and write the module source to its stdin. Even
with persistent ssh connections enabled, it's a noticeable improvement
to make only one ssh connection per task.
Unfortunately, pipelining is disabled by default because it is
incompatible with sudo's
requiretty setting (or su, which always requires a tty). This
is because of
a quirk of the Python interpreter,
which enters interactive mode automatically when you pipe in data from a
Update 2015-11-18: I've submitted a pull request to
make pipelining work with requiretty.
The rest of this post still remains true, but if the PR is merged, the
underlying problem will just go away.
Pipelining can be enabled globally by setting “pipelining=True”
in the ssh section of ansible.cfg, or setting
“ANSIBLE_SSH_PIPELINING=1” in the environment.
With Ansible 2 (not yet released), you can also
set ansible_ssh_pipelining in the inventory or in a playbook.
You can leave it enabled in ansible.cfg, but turn it off for
some hosts (where requiretty must remain enabled), or even
write a play with pipelining disabled in order to remove
requiretty from /etc/sudoers.
line: 'Defaults requiretty'
The above lineinfile recipe is simplistic, but it shows that
it's now possible to disable requiretty, even if it's by
replacing /etc/sudoers altogether.
Note the use of another Ansible 2 feature above: vars can also
be set for individual tasks (and
not only plays.
The ability to use “jump hosts” with Ansible is another often-requested
feature. This has been discussed repeatedly on the
has had a number of
written about it, and
have been submitted as pull requests to Ansible.
The recommended solution was to set a ProxyCommand in
~/.ssh/config. This meant duplicating inventory data and
keeping two sources of connection information in sync. It worked, but
grew rapidly less manageable with a larger inventory. Similarly, the
ssh_config inventory plugin was a makeshift solution at best.
This post describes the general mechanism provided in Ansible 2 (not yet
released) to make SSH configuration changes—including jump hosts—without
depending on any data external to Ansible.
The ssh_args setting in the ssh_connection section
of ansible.cfg is a global setting whose contents are prepended
to every command-line for ssh/scp/sftp. This behaviour has been retained
unmodified for backwards compatibility, but I don't recommend its use,
because it overrides the default persistence settings.
In addition to the above, the new ansible_ssh_common_args
inventory variable is appended to every command-line for
ssh/scp/sftp. This can be set in the inventory (for a group or a host)
or in a playbook (for a play, or block, or task). This is the place to
configure any ProxyCommand you want to use.
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p email@example.com"'
In addition to that, the new ansible_ssh_extra_args
variable is appended only to command-lines for ssh. There are analogous
ansible_scp_extra_args and ansible_sftp_extra_args
variables to change scp and sftp command-lines. This allows you to do
truly odd things like open a reverse-tunnel to the control node with
-R (which is an option only ssh accepts, not scp or sftp).
The --ssh-common-args command-line option is useful when
debugging (there's also --ssh-extra-args,
--scp-extra-args, and --sftp-extra-args). Note that
any values you set on the command-line will be overriden by the
inventory or playbook settings described above (which seems backwards,
but that's how Ansible handles other command-line options too).
Also note that
ansible_user, ansible_host, and ansible_port
are now preferred to the old ansible_ssh_* versions.
Once again, the modest user-visible changes are accompanied by major
changes internally. The
SSH connection plugin was rewritten
to be more maintainable, and an entire class of “my connection just
hangs” and other bugs (especially around privilege escalation) were
fixed in the process.