Managing Ansible through IAP
To begin with
So IAP is a great tool to help protect access to resources in your GCP environment. And ansible is a great low-barrier-to-entry tool to manage machines in your environment. But the two systems don’t really play well together. Work has been done to make the two tools talk better, but what existed didn’t work quite right when I tried it. So here’s a few updates…
How it all works
The inventory file
Ansible can create inventories from GCP API calls. So we first create an inventory file requesting host information from all our projects:
plugin: gcp_compute
projects:
- project1
- project2
auth_kind: application
keyed_groups:
- key: labels
prefix: label
- key: project
prefix: project
compose:
ansible_host: name
The link has more documentation on specific sections. The big thing that matters here is the keyed_groups
and compose
step. keyed_groups
will matter when you go to select groups of hosts, compose
matters for initial connections, so probably don’t mess with it.
The ansible.cfg file
To get this working nicely for users (so they don’t have to execute weird ansible commands) we need to set some defaults in a local ansible.cfg
file:
[inventory]
enable_plugins = gcp_compute
[defaults]
inventory = inventory/gcp.yml
[ssh_connection]
# Enabling pipelining reduces the number of SSH operations required
# to execute a module on the remote server.
# This can result in a significant performance improvement
# when enabled.
pipelining = True
ssh_executable = misc/gcp-wrapper.sh
ssh_args =
# Tell ansible to use SCP for file transfers when connection is set to SSH
scp_if_ssh = True
scp_executable = misc/gcp-scp-wrapper.sh
scp_args =
It’s very important that we leave the ssh_args =
and scp_args =
lines in, to ensure that ansible doesn’t try and add some weird additional arguments and break our IAP request (which doesn’t support any default SSH CLI switches like -O
).
The scripts
To actually leverage IAP, we need to call it correctly. That’s where our SSH/SCP override scripts come in:
gcp-wrapper.sh
#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP option to connect
# to our servers.
# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
# SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.
host="${*: -2: 1}"
cmd="${*: -1: 1}"
# Unfortunately ansible has hardcoded scp options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for s_arg in "${@: 1: $# -2}" ; do
if [[ "${s_arg}" == --* ]] ; then
opts+=("${s_arg}")
fi
done
exec gcloud compute ssh "${opts[@]}" "${host}" --command "${cmd}"
gcp-scp-wrapper.sh
#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP option to connect
# to our servers.
# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
# SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.
host="${*: -2: 1}"
cmd="${*: -1: 1}"
# Unfortunately ansible has hardcoded scp options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for s_arg in "${@: 1: $# -2}" ; do
if [[ "${s_arg}" == --* ]] ; then
opts+=("${s_arg}")
fi
done
# Remove [] around our host, as gcloud scp doesn't understand this syntax
cmd=$(echo "${cmd}" | tr -d "[]")
exec gcloud compute scp "${opts[@]}" "${host}" "${cmd}"
These two scripts will actually make the proper ssh/scp connections to your hosts. You can debug them by adding a set -x
to the script and running ansible
/ansible-playbook
with -vvv
.
The group vars
The final step to making this all work the way we want is actually defining our full IAP commands for each host. Fortunately, we can do this dynamically with group variables.
group_vars/all.yml
ansible_ssh_common_args: "--tunnel-through-iap --zone={{ hostvars[inventory_hostname].zone }} --project={{ hostvars[inventory_hostname].project }}"
ansible_scp_extra_args: "--tunnel-through-iap --zone={{ hostvars[inventory_hostname].zone }} --project={{ hostvars[inventory_hostname].project }}"
ansible_become: yes
ansible_become_user: root
ansible_become_method: sudo
And there’s our last bit of magic. We define the zone and project for each host we connect to using the inventory data for the hosts. Nice.
Results
We end up with an Ansible set up that lets us connect through IAP to hosts just like we would using SSH:
#$ ansible-playbook -D common.yml --ask-vault-pass
Vault password:
/usr/lib/python3.10/site-packages/google/auth/_default.py:79: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/usr/lib/python3.10/site-packages/google/auth/_default.py:79: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
PLAY [install common] *******************************
TASK [Gathering Facts] ******************************************