Our API gateway authenticates partners with client certificates. Each partner's CA certificate is a file in /etc/gateway/trusted on six gateway servers, and the reload step joins every file in that directory into the bundle nginx reads. When a partner leaves, a playbook removes their files and reloads the gateways. In May our contract with a regional carrier ended, and the offboarding run removed them, as far as anybody could see.

In October their old tracking system called our booking API. The account check refused it further in, but the TLS handshake had accepted their certificate, and the access log showed the same client certificate calling twice a day since May.

The task that should have removed it looked like this:

  • name: Remove retired partner certificates ansible.builtin.file: path: "{{ item }}" state: absent with_fileglob:
    • "/etc/gateway/trusted/{{ retired_partner }}-*.pem"

fileglob is a lookup, and lookups run on the Ansible controller, not on the hosts the play targets. Its documentation says so, and says that become and delegate_to make no difference. Our controller is a CI runner, which has no /etc/gateway at all. The pattern matched nothing, the loop had no items, and Ansible reported the task as skipping on each of the six servers. The recap showed six skips and no failures, and a skip is exactly what a removal task looks like when there is nothing left to remove.

The task had been written years earlier by someone who ran Ansible from one of the gateways, where the controller and the target were the same machine and the glob found the files. Moving the playbook into CI changed where it looked and not a word of what it said.

The files are now found on the servers themselves. ansible.builtin.find runs with paths set to /etc/gateway/trusted and patterns set to the partner's prefix, registers its result, and the file task loops over the paths it returned. After the reload a second find runs, and the play fails if it finds anything. A check in our pipeline rejects fileglob in any task that is not copying files from the controller, and every offboarding now ends with a test connection using the retired certificate, which has to be refused.

A glob in a playbook reads like a path on the server. It was a path on whichever machine ran Ansible, and that machine had none of our certificates.

– Sergey Shinder