Deploy a custom CA for my private registry

To be able to use your images that are stored in a secured private registry but skipping the TLS verification using the insecure-registry option is not acceptable, then you need to set the certificate on each worker node.

To accomplish this task, we provide a Kubernetes solution using privileged containers orchestrated by a daemonset. The following configuration contains three resources.

• A Configmap that is used to define the certificate and the registry host.
• A Configmap defining the script that writes the registry configuration to the worker node.
• A Daemonset that deploys the pod on every worker.

Important Notes:

• This section is only compatible with Containerd runtime.
• Keep in mind that this feature is a temporary solution. An equivalent solution available from the UI portal is in a development phase.

Please use the following solution as a base configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-registry
data:
  ca.crt: |+
    -----BEGIN CERTIFICATE-----
    MIIEtjCCAp4CCQChxj4cXfIoZzANBgkqhkiG9w0BAQsFADAdMRswGQYDVQQDDBJj
    WjAdMRswGQYDVQQDDBJjYWFzY29udGFpbmVyZHRlc3QwggIiMA0GCSqGSIb3DQEB
    AQUAA4ICDwAwggIKAoICAQClOk+HeRZkGondFTIAUmzEUZ/gKdwOwChiMrzJzWsI
    NL6MeMnF+Goj+5GaK6kejLTd35e+YEtHMthTiOlr+l+MbLDADP/NDUnBt0/zoA55
    1zPrLkEyr5aW2tCyrMUVL1rHvuBB3vhBFoeVwcHuLYMY+v6cUfSrMyBeZH20IoRq
    qv+r6+YJX3B8OQ3Ndwkxxzl02Yxbv7/6OA4NxrPiUfhSgGnzYxul7X7UzxToE1nQ
    /L2SwCB0zXL+Ww==
    -----END CERTIFICATE-----
  hostname: private.registry.com


---
apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-private-registry-script
data:
  setup.sh: |
    #!/bin/sh
    set -eu pipefail
    DIR=/privileged/etc/containerd/certs.d/$TRUSTED_REGISTRY
    mkdir -p $DIR
    echo '$TRUSTED_CERT' > $DIR/ca.crt
    cat > $DIR/hosts.toml << EOF
    server = "https://${TRUSTED_REGISTRY}"
 
    [host."https://${TRUSTED_REGISTRY}"]
      capabilities = ["pull", "resolve"]
      ca = ca.crt
    EOF
 
    echo "Registry added"
    #(Optional - Enable according to your needs)
    #echo "Adding trusted CA at OS level"
    #echo "$TRUSTED_CERT" > /privileged/usr/local/share/ca-certificates/ca.crt && chroot privileged update-ca-certificates


---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-configurator
spec:
  selector:
    matchLabels:
      k8s-app: node-configurator-pod
  template:
    metadata:
      labels:
        k8s-app: node-configurator-pod
    spec:
      imagePullSecrets:
      - name: regcred2
      hostIPC: true
      hostPID: true
      hostNetwork: true
      initContainers:
      - name: registry-addition-service
        command: ["/scripts/setup.sh"]
        image: cnbb-docker-local.bin.swisscom.com/alpine:3.17
        env:
        - name: TRUSTED_CERT
          valueFrom:
            configMapKeyRef:
              name: trusted-registry
              key: ca.crt
        - name: TRUSTED_REGISTRY
          valueFrom:
            configMapKeyRef:
              name: trusted-registry
              key: hostname
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true
          runAsUser: 0
          capabilities:
            add: ["*"]
        volumeMounts:
        - name: setup-script
          mountPath: /scripts
        - name: root-host
          mountPath: /privileged
      containers:
      - name: sleeping-container
        command: ["sleep"]
        args: ["3000"]
        image: cnbb-docker-local.bin.swisscom.com/alpine:3.17
      volumes:
      - name: setup-script
        configMap:
          name: setup-private-registry-script
          defaultMode: 0744
      - name: root-host
        hostPath:
          path: /
          type: ""

Good to know:

• With containerd v1.5, new and additional registry hosts config support has been implemented for several clients and especially for CRI clients. Hence, with this new config, updates under the /etc/containerd/certs.d/ directory do not require restarting the containerd daemon.
• Feel free to use any sleeping image to keep your pod alive after the init-containers finished their job.

You could read more about the containerd registry customization here.