GKE and private registries with Self Signed Certificates

NB: This article provides a workaround, this is not a Google Cloud officially documentation procedure, use at your own risk and only on non-production systems

What problem we are trying to solve?

GKE cannot pull images from a registry that uses certificates that are not signed by a trusted CA: if the kubelet on the node is not able to verify the CA authority for the registry it’s trying to pull an image from, the application pod is stuck in the ContainerCreating stage with an error like “X509: certificate signed by unknown authority”. This behavior is in place for security purposes and cannot be changed.

This is quite important as some people want to stand up a private Docker registry for rapid prototyping or use a free trial version of something like Jfrog Artifactory which deploys with Self Signed Certificates. And you don’t always have time to do things properly, especially if you are evaluating the registry you want to use.

This guide shows a workaround for this issue. We will explore GKE with both Containerd and Docker runtimes. Note that starting from GKE 1.19, the default image when creating a cluster is Container-Optimized OS with Containerd unless you pick a different OS/runtime at cluster creation.

How to know what runtime my cluster is using?

Run the following command against your cluster, replace CLUSTER_NAME with your cluster

gcloud container node-pools list \
    --cluster CLUSTER_NAME \
    --format="table(name,version,config.imageType)"

The output will look something like

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.20.9-gke.700  COS_CONTAINERD
pool-1        1.20.9-gke.700  COS_CONTAINERD

How does the workaround work?

This workaround uses DaemonSets to upload the necessary certificate files into each node in the cluster, DaemonSets are resource types in Kubernetes that ensures each node runs exactly one copy of the pod. They run when the node is marked as Ready, if the node is restarted (due to an upgrade for example) the Pod will be executed again.

The workaround documented here consists of running a privileged DaemonSet (basically a pod with root access on the node) that will upload files to a specific part of the filesystem and restart the runtime daemon if needed). Following Google best practices, which consist of having the shell script in a ConfigMap instead of directly in the DaemonSet, allows us to format the script properly and make it easy to read.

What the DaemonSet does on the node depends on the runtime. Below we have the procedures for both Docker and Containerd.

But regardless of which runtime is used, you need to upload the Registry certificate as a secret to GKE and also the credentials for accessing the registry. Below I’m using an example with Username/Password.

Upload the private certificate as a Secret.

The first step is to make the self-signed certificate available in GKE as a secret, using the kubectl CLI and the .pem or .crt file, run the following command.

kubectl create secret generic registry-pem — from-file=cert.pem

Create a Docker secret with the username/password of the private registry

The next step is to configure the credentials needed by GKE to authenticate toward the registry. I used Jfrog Artifactory for my tests and it allows authentication via username/password. You will need to create a secret with the URL, username, and password of the registry and append it to the Deployments:

kubectl create secret docker-registry regsecret \
--docker-server=myregistry.mycompany.com \
--docker-username=USERNAME \
--docker-password=PASSWORD

You can then reference it in the deployment like this:

Docker based clusters

Docker supports hot-loading certificates without restarting the Docker daemon, as long as they are located in the proper folder. For each private registry, you want the GKE cluster to pull images from, create a folder with the same FQDN and port of the registry under /etc/docker/certs.d, for example, if the registry FQDN is registry.mycompany.com and port is 5000 you need to create a folder like this /etc/docker/certs.d/registry.mycompany.com:5000

• The Certs.d folder doesn’t exist by default so it needs to be created as well.
• If your registry serves traffic on 443 there is no need to add the port to the folder name. Docker assumes HTTPS.
• The file needs to have the extension .crt.

The following ConfigMap and DaemonSets can be used to upload the certificate to each node.

Replace the REGISTRY_FQDN variable in the ConfigMap with the URL of your Registry. If the registry is serving on a custom port (NOT on 443) append it to the URL with a column (:).

Apply the ConfigMap than the DaemonSet

kubectl apply -f https://raw.githubusercontent.com/boredabdel/gke-private-registries/main/docker/script.yaml
kubectl apply -f https://raw.githubusercontent.com/boredabdel/gke-private-registries/main/docker/cert-ds.yaml

Check the DaemonSets are running, they should take few seconds.

kubectl get pods

You can optionally SSH into one of the nodes in the cluster and verify that the folder has been created and that the certificates are uploaded to the node. Check the troubleshooting section if you are facing issues.

Containerd based clusters

Contained requires restarting the Daemon. It also supports customizing the Daemon to pull images from registries with self-signed certificates. You need to edit the /etc/containerd/config.toml file and add the following configuration:

The following ConfigMap and DaemonSet below contains a script that will:

• Extract the certificate from a secret and write it to /etc/custom_certs/cert.pem
• Check if the Container config already has the needed config, if not append it.
• Restart Containerd.

Make sure to change the following configs in the ConfigMap before applying these manifests

• Replace registry.mycompany.com with the FQDN of your registry, appending the port if it’s not 443 at the end.
• Populate the REGISTRY_FQDN variable with the FQDN of your registry appending the port as needed.

Troubleshooting

To check what the DaemonSet is doing you need to check the logs of the init container. Grab the pod name and run

kubectl logs POD_NAME -c init

The code can be found in this repo