A month or two ago I noticed that an application I was working on broke after an automatic terraform apply that had been ran as part of a CI job. kubectl quickly revealed an ImagePullBackOff as the culprit. It turned out that the wrong image tag had been specified in one of the .tf files and the kubelet was trying to pull a non-existent image. While it would be nice if the Kubernetes Terraform provider was smart enough to check for this ahead of time it makes sense as to why it doesn’t… your machine isn’t on the same network as the kubelet, isn’t using the same credentials for the Docker pull, and might not be running the same OS or container engine. To remedy this I decided to whip up a small script that extracts the images necessary for a successful terraform apply from a terraform plan output:
images=$(terraform plan | grep "\\s*[~]\s*image.*->\s*.*" | tr -s ' ' | rev | cut -d ' ' -f 1 | rev | tr -d '"')
This produces a newline-delimited list of qualified Docker images and tags. From here you can either use an experimental Docker command to check for the existence of each image…
docker login -u jack_burton -p porkchop_express
for i in $images
do
echo Checking for existence of $i
docker manifest inspect $i
done
…or use some curl magic to query the registry the hard way (well, easy for you since I did the API poking for you with a bit of help from Dr. Internets MD):
token=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "jack_burton", "password": "porkchop_express" https://hub.docker.com/v2/users/login/ | jq -r .token)
for i in $images
do
image=$(echo $i | cut -d : -f 1)
tag=$(echo $i | cut -d : -f 2)
echo Checking for existence of $i
curl -f --head -lL -H "Authorization: Bearer ${token}" https://hub.docker.com/v2/repositories/$image/tags/$tag/ > /dev/null
done
Method 1 is obviously “easier” at the expense of requiring the Docker CLI in your CI runtime environment whereas method 2 needs nothing but good ol' curl and old-fashioned coreutils magic. Pick your poison and revel in the fact that your automatic terraform apply operations are now that much safer!