Verifying signatures for custom images

You can sign your apps' images and restrict the cluster from running images without a valid signature. When creating a pod, the Policy Manager component based on Kyverno verifies the signature. The Cosign utility creates signatures.

In this guide, you will create a pair of keys, sign you image, apply a policy, and ensure the inability to run an unsigned image.

Getting startedGetting started

1. Install Cosign version 3 or newer and crane.

   Commands in this guide were tested with Cosign v3.1.3.

2. Make sure you have write permissions for the registry with your images: the signature is saved to the same repository as the image.

3. Make sure the registry is accessible from the cluster. Kyverno inside the cluster verifies the signature, so it accesses the registry on its own, regardless of how the node loads the image.

4. Make sure that Policy Manager is enabled:

   kubectl get policymanagerconfig main -o jsonpath='{.spec.enabled}'
   

   If the command returns false, enable the component (see Activating a policy preset).

How it worksHow it works

Cosign saves the signature next to the image: as a separate tag in sha256-<digest>.sig format in the same repository. When a pod is created in the cluster, Kyverno intercepts the request, downloads the signature from the registry, and verifies it using your public key. If the signature is missing or mismatches the key, the pod is not created.

The verification uses the image digest, not the tag. Therefore, you need to sign the image using the digest as well: it is possible to overwrite a tag, but not a digest.

Step 1: Generate a key pairStep 1: Generate a key pair

1. Set a password for the private key and create a pair:

   export COSIGN_PASSWORD='<private key password>'
   cosign generate-key-pair
   

   As a result, you will have two files:

   • cosign.key: Private key you sign images with.
   • cosign.pub: Public key Kyverno verifies signatures with.

Step 2: Sign the imageStep 2: Sign the image

1. Get the image digest:

   IMAGE_REPOSITORY='<registry address>/<image path>'
   IMAGE_TAG='<tag>'
   DIGEST=$(crane digest "${IMAGE_REPOSITORY}:${IMAGE_TAG}")
   IMAGE="${IMAGE_REPOSITORY}@${DIGEST}"
   

2. Sign the image:

   cosign sign --yes \
     --key cosign.key \
     --use-signing-config=false \
     --new-bundle-format=false \
     --tlog-upload=false \
     "$IMAGE"
   

   The flags serve the following purposes:

   • --tlog-upload=false: Not to send a record to an external Rekor log.
   • --use-signing-config=false: Not to use a configuration of Sigstore services. Without this flag, Cosign version 3 forbids --tlog-upload=false.
   • --new-bundle-format=false: To store the signature as a separate tag in sha256-<digest>.sig format. By default, Cosign version 3 publishes the signature as an OCI 1.1 artifact, which Kyverno of this version cannot read, and rejects some registries.

   Cosign will show warnings that --new-bundle-format and --tlog-upload are declared obsolete. This is expectable: both flags are required as long as Kyverno keeps working with the previous signature format.

3. Make sure the signature appears in the registry:

   crane ls "$IMAGE_REPOSITORY" | grep '^sha256-'
   

   The output should contain the sha256-<digest>.sig tag, with <digest> matching the signed image's digest.

Step 3: Verify the signature locallyStep 3: Verify the signature locally

Verify the signature before configuring the policy: this way, you can separate the signature error from the policy error:

cosign verify --key cosign.pub --insecure-ignore-tlog "$IMAGE"

For a signed image, the command will show the verification result, while for an unsigned one, the no signatures found error.

Step 4: Grant Kyverno access to the registryStep 4: Grant Kyverno access to the registry

If your registry requires authentication, create a secret with credentials:

kubectl create secret docker-registry my-registry-creds \
  --namespace stackland-policy-manager \
  --docker-server=<registry address> \
  --docker-username=<username> \
  --docker-password=<password or token>

Step 5: Apply the policyStep 5: Apply the policy

The manifest below uses report safe mode: failureAction: Audit, mutateDigest: false, and webhookConfiguration.failurePolicy: Ignore. Once applied, the policy registers violations, but does not block pod creation. You will enable the blocking gradually after verifying the policy and registry access.

1. Create the ClusterPolicy resource file, e.g., using the touch clusterpolicy.yaml command.

2. Open the file and paste the configuration below into it:

   apiVersion: kyverno.io/v1
   kind: ClusterPolicy
   metadata:
     name: verify-my-images
   spec:
     admission: true
     background: true
     webhookConfiguration:
       failurePolicy: Ignore
     rules:
     - name: verify-signature
       match:
         any:
         - resources:
             kinds:
             # Pod only: Kyverno generates rules for Deployment, StatefulSet, DaemonSet, Job, and CronJob.
             # Adding these kinds here disables autogeneration.
             - Pod
             # namespaceSelector:
             #   matchLabels:
             #     stackland.yandex.cloud/project-name: <project-name>
       exclude:
         any:
         - resources:
             namespaceSelector:
               matchLabels:
                 # Platform namespaces are verified on the node, not by this policy.
                 stackland.yandex.cloud/project-name: stackland
       verifyImages:
       - imageReferences:
         # Match only customer-owned paths. Never use * here.
         - <registry address>/<path to your images>/*
         failureAction: Audit
         type: Cosign
         mutateDigest: false
         verifyDigest: true
         required: true
         useCache: true
         imageRegistryCredentials:
           secrets:
           # The Secret must exist in the stackland-policy-manager namespace.
           - my-registry-creds
         attestors:
         - count: 1
           entries:
           - keys:
               publicKeys: |-
                 -----BEGIN PUBLIC KEY-----
                 <contents of cosign.pub>
                 -----END PUBLIC KEY-----
               signatureAlgorithm: sha256
               rekor:
                 ignoreTlog: true
               ctlog:
                 ignoreSCT: true
   

3. Provide the following in the parameter:

   • spec.rules[0].verifyImages[0].imageReferences: Paths to your images.
   • spec.rules[0].verifyImages[0].imageRegistryCredentials.secrets: Secret name from Step 4. If the registry does not require authentication, delete the imageRegistryCredentials section.
   • spec.rules[0].verifyImages[0].attestors[0].entries[0].keys.publicKeys: Contents of the cosign.pub file.

   By default, the policy verifies images in all namespaces of all projects, except for the platform namespaces. The exclude section with the stackland.yandex.cloud/project-name: stackland label excludes the platform namespaces regardless of their names.

   To check images only in namespaces within the same projects, uncomment the namespaceSelector section under match and specify the project name:

   match:
     any:
     - resources:
         kinds:
         - Pod
         namespaceSelector:
           matchLabels:
             stackland.yandex.cloud/project-name: <project name>
   

4. Apply the policy in report mode:

   kubectl apply -f clusterpolicy.yaml
   

5. Make sure the policy is ready:

   kubectl get clusterpolicy verify-my-images -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
   

   Expected result: True.

Enabling blocking graduallyEnabling blocking gradually

The policy is already active in report safe mode. Before enabling blocking, make sure it covers only the relevant images and successfully accesses the registry.

1. Make sure there are no unexpected violations under System > Events in the management console. At the same time, enable blocking of unsigned images and replacement of tags with a digest. Registry access errors will remain unblocking for now:

   kubectl patch clusterpolicy verify-my-images --type=json --patch='[
     {
       "op": "replace",
       "path": "/spec/rules/0/verifyImages/0/failureAction",
       "value": "Enforce"
     },
     {
       "op": "replace",
       "path": "/spec/rules/0/verifyImages/0/mutateDigest",
       "value": true
     }
   ]'
   

2. Make sure that attempts to access the registry are stable and enable blocking in case of verification errors:

   kubectl patch clusterpolicy verify-my-images --type=json --patch='[
     {
       "op": "replace",
       "path": "/spec/webhookConfiguration/failurePolicy",
       "value": "Fail"
     }
   ]'
   

   The policy will switch to the final mode, Enforce, and then to Fail.

Step 6: Test the policyStep 6: Test the policy

In the commands below, <namespace name> is the name of a specific namespace from the policy scope, e.g., team-alpha-backend.

1. Run a pod with a signed image:

   kubectl run test-signed -n <namespace name> --restart=Never \
     --image=<registry address>/<image path>:<tag>
   

   This will create a pod. Make sure that Kyverno has replaced the tag with the digest and marked the verification with an annotation:

   kubectl get pod test-signed -n <namespace name> \
     -o jsonpath='{.spec.containers[0].image}{"\n"}{.metadata.annotations.kyverno\.io/verify-images}{"\n"}'
   

   Here is an example of the command output:

   registry.example.com/myteam/app:1.0.0@sha256:7c38f24774e3cbd906d2d33c38354ccf787635581c122965132c9bd309754d4a
   {"registry.example.com/myteam/app@sha256:7c38f24774e3cbd906d2d33c38354ccf787635581c122965132c9bd309754d4a":"pass"}
   

2. Run a pod with an unsinged image:

   kubectl run test-unsigned -n <namespace name> --restart=Never \
     --image=<registry address>/<image path>:<unsigned image tag>
   

   This will not create a pod.

   Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
   
   resource Pod/<namespace name>/test-unsigned was blocked due to the following policies
   
   verify-my-images:
     verify-signature: 'failed to verify image registry.example.com/myteam/app:unsigned:
       .attestors[0].entries[0].keys: no signatures found'
   

3. Delete the test pod.

   kubectl delete pod test-signed -n <namespace name>
   

LimitationsLimitations

• Registry becomes a dependency when creating pods. In failurePolicy: Fail mode, registry unavailability and expired credentials stop the creation of pods with images covered by the policy. The verification cache and narrow list of imageReferences limit the effects, but do not eliminate them.
• The policy is deleted together with the component. If you disable Policy Manager (PolicyManagerConfig.spec.enabled: false), the policy and the secret with registry credentials will be deleted. To re-enable it, recreate them.