Sigstore

Verify Sigstore Cosign format signatures and attestations using keys, certificates, or keyless attestors.

Sigstore is a Linux Foundation project focused on software signing and transparency log technologies to improve software supply chain security. Cosign is a sub-project that provides image signing, verification, and storage in an OCI registry.

Verifying Image Signatures

Container images can be signed during the build phase of a CI/CD pipeline using the Cosign CLI. An image can be signed with multiple signatures, for example at the organization level and at the project level.

The policy rule check fails if the required signatures are not found in the OCI registry, or if the image was not signed using matching attestors.

The rule mutates matching images to add the image digest, when mutateDigest is set to true (which is the default), if the digest is not already specified. Using an image digest has the benefit of making image references immutable. This helps ensure that the version of the deployed image does not change and, for example, is the same version that was scanned and verified by a vulnerability scanning and detection tool.

Here is a sample image verification policy which ensures an image from the ghcr.io/kyverno/test-verify-image repository, using any tag, is signed with the corresponding public key as defined in the policy:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image
 5spec:
 6  validationFailureAction: Enforce
 7  background: false
 8  webhookTimeoutSeconds: 30
 9  failurePolicy: Fail
10  rules:
11    - name: check-image
12      match:
13        any:
14        - resources:
15            kinds:
16              - Pod
17      verifyImages:
18      - imageReferences:
19        - "ghcr.io/kyverno/test-verify-image*"
20        attestors:
21        - count: 1
22          entries:
23          - keys:
24              publicKeys: |-
25                -----BEGIN PUBLIC KEY-----
26                MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
27                5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
28                -----END PUBLIC KEY-----

Note

The public key may either be defined in the policy directly or reference a standard Kubernetes Secret elsewhere in the cluster by specifying it in the format k8s://<namespace>/<secret_name>. The named Secret must specify a key cosign.pub containing the public key used for verification. Secrets may also be referenced using the secret{} object. See kubectl explain clusterpolicy.spec.rules.verifyImages.attestors.entries.keys for more details on the supported key options.

A signed image can be run as follows:

1kubectl run signed --image=ghcr.io/kyverno/test-verify-image:signed
2pod/signed created

The deployed Pod will be mutated to use the image digest.

Attempting to run an unsigned image will produce a policy error as follows:

1kubectl run unsigned --image=ghcr.io/kyverno/test-verify-image:unsigned
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/unsigned was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:unsigned:
8    signature not found'

Similarly, attempting to run an image which matches the specified rule but is signed with a different key will produce an error:

1kubectl run signed-other --image=ghcr.io/kyverno/test-verify-image:signed-by-someone-else
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/signed-other was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:signed-by-someone-else:
8    invalid signature'

Signing images

To sign images, install Cosign and generate a public-private key pair.

1cosign generate-key-pair

Next, use the cosign sign command and specifying the private key in the --key command line argument.

1# ${IMAGE} is REPOSITORY/PATH/NAME:TAG
2cosign sign --key cosign.key ${IMAGE}

This command will sign your image and publish the signature to the OCI registry. You can verify the signature using the cosign verify command.

1cosign verify --key cosign.pub ${IMAGE}

Refer to the Cosign documentation for usage details and OCI registry support.

Verifying Image Attestations

Container image signatures prove that the image was signed by the holder of a matching private key. However, signatures do not provide additional data and intent that frameworks like SLSA (Supply chain Levels for Software Artifacts) require.

An attestation is metadata attached to software artifacts like images. Signed attestations provide verifiable information required for SLSA.

The in-toto attestation format provides a flexible scheme for metadata such as repository and build environment details, vulnerability scan reports, test results, code review reports, or any other information that is used to verify image integrity. Each attestation contains a signed statement with a predicateType and a predicate. Here is an example derived from the in-toto site:

 1{
 2  "payloadType": "https://example.com/CodeReview/v1",
 3  "payload": {
 4    "_type": "https://in-toto.io/Statement/v0.1",
 5    "predicateType": "https://example.com/CodeReview/v1",
 6    "subject": [
 7      {
 8        "name": "registry.io/org/app",
 9        "digest": {
10          "sha256": "b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105"
11        }
12      }
13    ],
14    "predicate": {
15      "author": "alice@example.com",
16      "repo": {
17        "branch": "main",
18        "type": "git",
19        "uri": "https://git-repo.com/org/app"
20      },
21      "reviewers": [
22        "bob@example.com"
23      ]
24    }
25  },
26  "signatures": [
27    {
28      "keyid": "",
29      "sig": "MEYCIQDtJYN8dq9RACVUYljdn6t/BBONrSaR8NDpB+56YdcQqAIhAKRgiQIFvGyQERJJYjq2+6Jq2tkVbFpQMXPU0Zu8Gu1S"
30    }
31  ]
32}

The imageVerify rule can contain one or more attestation checks that verify the contents of the predicate. Here is an example that verifies the repository URI, the branch, and the reviewers.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: attest-code-review
 5spec:
 6  validationFailureAction: Enforce
 7  background: false
 8  webhookTimeoutSeconds: 30
 9  failurePolicy: Fail
10  rules:
11    - name: attest
12      match:
13        any:
14        - resources:
15            kinds:
16              - Pod
17      verifyImages:
18      - imageReferences:
19        - "registry.io/org/app*"
20        attestations:
21          - predicateType: https://example.com/CodeReview/v1
22            attestors:
23            - entries:
24              - keys:
25                  publicKeys: |-
26                    -----BEGIN PUBLIC KEY-----
27                    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEzDB0FiCzAWf/BhHLpikFs6p853/G
28                    3A/jt+GFbOJjpnr7vJyb28x4XnR1M5pwUUcpzIZkIgSsd+XcTnrBPVoiyw==
29                    -----END PUBLIC KEY-----                    
30            conditions:
31              - all:
32                - key: "{{ repo.uri }}"
33                  operator: Equals
34                  value: "https://git-repo.com/org/app"            
35                - key: "{{ repo.branch }}"
36                  operator: Equals
37                  value: "main"
38                - key: "{{ reviewers }}"
39                  operator: AnyIn
40                  value: ["ana@example.com", "bob@example.com"]

The policy rule above fetches and verifies that the attestations are signed with the matching private key, decodes the payloads to extract the predicate, and then applies each condition to the predicate.

Each verifyImages rule can be used to verify signatures or attestations, but not both. This allows the flexibility of using separate signatures for attestations. The attestors{} object appears both under verifyImages as well as verifyImages.attestations. Use of it in the former location is for image signature validation while use in the latter is for attestations only.

Note

The structure of the predicate may differ slightly depending on the predicate type used during attestation. Use cosign verify-attestation by passing the expected predicate type with the --type flag and examine the predicate structure to ensure the expression you’re writing is accurate.

Signing attestations

To sign attestations, use the cosign attest command. This command will sign your attestations and publish them to the OCI registry.

1# ${IMAGE} is REPOSITORY/PATH/NAME:TAG
2cosign attest --key cosign.key --predicate <file> --type <predicate type>  ${IMAGE}

You can use a custom attestation type with a JSON document as the predicate. For example, with the code review example above the predicate body can be specified in a file predicate.json with the contents:

 1{
 2    "author": "alice@example.com",
 3    "repo": {
 4        "branch": "main",
 5        "type": "git",
 6        "uri": "https://git-repo.com/org/app"
 7    },
 8    "reviewers": [
 9        "bob@example.com"
10    ]
11}

The following cosign command creates the in-toto format attestation and signs it with the specified credentials using the custom predicate type https://example.com/CodeReview/v1:

1cosign attest ghcr.io/jimbugwadia/app1:v1 --key <KEY> --predicate predicate.json  --type https://example.com/CodeReview/v1

This flexible scheme allows attesting and verifying any JSON document, including vulnerability scan reports and Software Bill Of Materials (SBOMs).

Attestations, such as the snippet shown above, are base64 encoded by default and may be verified and viewed with the cosign verify-attestation command. For example, the below command will verify and decode the attestation of type slsaprovenance for a given image which was signed with the keyless signing ability.

1COSIGN_EXPERIMENTAL=true cosign verify-attestation --type slsaprovenance registry.io/myrepo/myimage:mytag | jq .payload -r | base64 --decode | jq

Verification of an attestation using a public key can be done simply:

1cosign verify-attestation --key cosign.pub --type <type> ${IMAGE}

Refer to the Cosign documentation for additional details including OCI registry support.

Certificate based signing and verification

This policy checks if an image is signed using a certificate. The root certificate, and any intermediary certificates in the signing chain, can also be provided in the certChain declaration.

 1---
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: check-image
 6spec:
 7  validationFailureAction: Enforce
 8  rules:
 9    - name: verify-signature
10      match:
11        any:
12        - resources:
13            kinds:
14              - Pod
15      verifyImages:
16      - imageReferences:
17        - "ghcr.io/kyverno/test-verify-image:signed-cert"
18        attestors:
19        - entries:
20          - certificates:
21              cert: |-
22                -----BEGIN CERTIFICATE-----
23                MIIDuzCCAqOgAwIBAgIUDG7gFB8RMMOMGkDm6uEusOE8FWgwDQYJKoZIhvcNAQEL
24                BQAwbDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQwwCgYDVQQHDANTSkMxEDAO
25                BgNVBAoMB05pcm1hdGExEDAOBgNVBAMMB25pcm1hdGExHjAcBgkqhkiG9w0BCQEW
26                D2ppbUBuaXJtYXRhLmNvbTAeFw0yMjA0MjgxOTU0NDFaFw0yNDA3MzExOTU0NDFa
27                MGwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEMMAoGA1UEBwwDU0pDMRAwDgYD
28                VQQKDAdOaXJtYXRhMRAwDgYDVQQDDAduaXJtYXRhMR4wHAYJKoZIhvcNAQkBFg9q
29                aW1AbmlybWF0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDP
30                LObWc4VM4CULHrjScdjAXwdeJ6o1SwS9Voz9wTYAASp54EDqgzecWGjtn409NF9o
31                4tqd5LotEFscoMXGpmm7dBpv76MQhGym7JBhlYaBksmnKp17nTfAmsgiDiUnjnG6
32                BQ5/FIdZYHtpJmMZ/SZqQ3ehXLaGj2qogPrEsObN1S/1b+0guLC/gVi1fiuUgd4Z
33                SDEmDaLjSuIQBrtba08vQnl5Ihzrag3A85+JNNxk9WBDFnLHMsRvlrUMU4565FS9
34                X57epDZakKvLATAK0/gKI2ZvWfY0hoO3ngEk4Rkek6Qeh1vXFBc8Rsym8W0RXjux
35                JDkye5RTsYrlXxSavP/xAgMBAAGjVTBTMB8GA1UdIwQYMBaAFBF3uwHovsxj7WxS
36                vDDKBTwuR+oaMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgTwMBgGA1UdEQQRMA+CDWhl
37                bGxmaXNoLnRlc3QwDQYJKoZIhvcNAQELBQADggEBAHtn9KptJyHYs45oTsdmXrO0
38                Fv0k3jZnmqxHOX7OiFyAkpcYUTezMYKHGqLdme0p2VE/TdQmGPEq1dqlQbF7UMb/
39                o+SrvFpmAJ1iAVjLYQ7KDCE706NgnVkxaPfU8UBOw2vF5nsgIxcheOyxTplbVOVM
40                vcYYwAWXxkNhrQ4sYygXuNgZawruxY1HdUgGWlh9XY0J5OBrXyinh2YGBUGQJgQR
41                NEmM+GQjdquPqAgDsb3kvWgFDrcbBZJBc/CyZU8GH9uIuPDgfVhDTqFtiz9W/F5s
42                Hh8yD7VAIWgL9TkGWRwWdD6Qx/BAu7dMdpjAxdGpMLn3O4SDAZDnQneaHx6qr/I=
43                -----END CERTIFICATE-----                
44              certChain: |-
45                -----BEGIN CERTIFICATE-----
46                MIIDuTCCAqGgAwIBAgIUU1kkhcMc+7ci1qvkLCre5lbH68owDQYJKoZIhvcNAQEL
47                BQAwbDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQwwCgYDVQQHDANTSkMxEDAO
48                BgNVBAoMB05pcm1hdGExEDAOBgNVBAMMB25pcm1hdGExHjAcBgkqhkiG9w0BCQEW
49                D2ppbUBuaXJtYXRhLmNvbTAeFw0yMjA0MjgxOTE2NTJaFw0yNzA0MjcxOTE2NTJa
50                MGwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEMMAoGA1UEBwwDU0pDMRAwDgYD
51                VQQKDAdOaXJtYXRhMRAwDgYDVQQDDAduaXJtYXRhMR4wHAYJKoZIhvcNAQkBFg9q
52                aW1AbmlybWF0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx
53                hpgJ/YUXtUyLNjJgoOBQHSIL6PrdNj9iemgddVg1WGzQrtMnleVY1Wh31C3nV2oN
54                VrcH2+i/14fyTWpAPEoJ/E6/3Pd8EYokFffm6AXvSCX6gaRpgeiWySK9T62bI7TP
55                4VplppF4lkUJbYxtFiVt5q2T4+lm+k8Q5kDtxU8d1067ApM82f9kHgoLqJwuuGM7
56                VPHX023orJ2YU68gJo78qGbv+1/aoPpcEZelk5RBXplvOT23DbMgEi3SxWjJ3djU
57                svQu+FMLG9xWpTdH5P98/1hY89xxYk+paEVDX0xSmINt2nfFGV5x1ChEMaZSC/7Q
58                9Z5qRX2e26/Mm+jFnIIJAgMBAAGjUzBRMB0GA1UdDgQWBBQRd7sB6L7MY+1sUrww
59                ygU8LkfqGjAfBgNVHSMEGDAWgBQRd7sB6L7MY+1sUrwwygU8LkfqGjAPBgNVHRMB
60                Af8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCGMBvR7wGHQdofwP4rCeXY9OlR
61                RamGcOX7GLI5zQnO717l+kZqJQAQfhgehbm14UkXx3/1iyqSYpNUIeY6XZaiAxMC
62                fQI8ufcaws4f522QINGNLQGzzt2gkDAg25ARTgH4JVmRxiViTsfrb+VgjcYhkLK5
63                mWffp3LpCiybZaRKwS93SNWo95ld2VzDgzGNLLGejifCe9nPSfvkuXHfDW9nSRMP
64                plXrFYd7TTMUaENRmTQtl1KyIlnLEp+A6ZBpY1Pxdc9SnflYQVQb0hsxSa+Swkb6
65                hRkMf01X7+GAI75hpgoX/CuCjd8J5kozsXLzUtKRop5gXyZxuFL8yUW9gfQs
66                -----END CERTIFICATE-----

To verify using the root certificate only, the leaf certificate declaration cert can be omitted.

 1---
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: check-image
 6spec:
 7  validationFailureAction: Enforce
 8  rules:
 9    - name: verify-signature
10      match:
11        any:
12        - resources:
13            kinds:
14              - Pod
15      verifyImages:
16      - imageReferences:
17        - "ghcr.io/kyverno/test-verify-image:signed-cert"
18        attestors:
19        - entries:
20          - certificates:
21              certChain: |-
22                -----BEGIN CERTIFICATE-----
23                MIIDuTCCAqGgAwIBAgIUU1kkhcMc+7ci1qvkLCre5lbH68owDQYJKoZIhvcNAQEL
24                BQAwbDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQwwCgYDVQQHDANTSkMxEDAO
25                BgNVBAoMB05pcm1hdGExEDAOBgNVBAMMB25pcm1hdGExHjAcBgkqhkiG9w0BCQEW
26                D2ppbUBuaXJtYXRhLmNvbTAeFw0yMjA0MjgxOTE2NTJaFw0yNzA0MjcxOTE2NTJa
27                MGwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEMMAoGA1UEBwwDU0pDMRAwDgYD
28                VQQKDAdOaXJtYXRhMRAwDgYDVQQDDAduaXJtYXRhMR4wHAYJKoZIhvcNAQkBFg9q
29                aW1AbmlybWF0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx
30                hpgJ/YUXtUyLNjJgoOBQHSIL6PrdNj9iemgddVg1WGzQrtMnleVY1Wh31C3nV2oN
31                VrcH2+i/14fyTWpAPEoJ/E6/3Pd8EYokFffm6AXvSCX6gaRpgeiWySK9T62bI7TP
32                4VplppF4lkUJbYxtFiVt5q2T4+lm+k8Q5kDtxU8d1067ApM82f9kHgoLqJwuuGM7
33                VPHX023orJ2YU68gJo78qGbv+1/aoPpcEZelk5RBXplvOT23DbMgEi3SxWjJ3djU
34                svQu+FMLG9xWpTdH5P98/1hY89xxYk+paEVDX0xSmINt2nfFGV5x1ChEMaZSC/7Q
35                9Z5qRX2e26/Mm+jFnIIJAgMBAAGjUzBRMB0GA1UdDgQWBBQRd7sB6L7MY+1sUrww
36                ygU8LkfqGjAfBgNVHSMEGDAWgBQRd7sB6L7MY+1sUrwwygU8LkfqGjAPBgNVHRMB
37                Af8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCGMBvR7wGHQdofwP4rCeXY9OlR
38                RamGcOX7GLI5zQnO717l+kZqJQAQfhgehbm14UkXx3/1iyqSYpNUIeY6XZaiAxMC
39                fQI8ufcaws4f522QINGNLQGzzt2gkDAg25ARTgH4JVmRxiViTsfrb+VgjcYhkLK5
40                mWffp3LpCiybZaRKwS93SNWo95ld2VzDgzGNLLGejifCe9nPSfvkuXHfDW9nSRMP
41                plXrFYd7TTMUaENRmTQtl1KyIlnLEp+A6ZBpY1Pxdc9SnflYQVQb0hsxSa+Swkb6
42                hRkMf01X7+GAI75hpgoX/CuCjd8J5kozsXLzUtKRop5gXyZxuFL8yUW9gfQs
43                -----END CERTIFICATE-----

This enables use cases where, in an enterprise with a private CA, each team has their own leaf certificate used for signing their images, and a global policy is used to verify all images signatures.

Signing images using certificates

To use certificates for image signing, you must first extract the public key using the cosign import command.

Assuming you have a root CA myCA.pem and a public-private certificate pair test.crt and test.key, you can convert the public certificate to a key as follows:

1cosign import-key-pair --key test.key

This creates the import-cosign.key and import-cosign.pub files. You can then sign using the certificate as follows:

1cosign sign $IMAGE --key import-cosign.key --cert test.crt --cert-chain myCA.pem

This image can now be verified using the leaf or root certificates.

Keyless signing and verification

The following policy verifies an image signed using ephemeral keys and signing data stored in a transparency log, known as keyless signing:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image-keyless
 5spec:
 6  validationFailureAction: Enforce
 7  webhookTimeoutSeconds: 30
 8  rules:
 9    - name: check-image-keyless
10      match:
11        any:
12        - resources:
13            kinds:
14              - Pod
15      verifyImages:
16      - imageReferences:
17        - "ghcr.io/kyverno/test-verify-image:signed-keyless"
18        attestors:
19        - entries:
20          - keyless:
21              subject: "*@nirmata.com"
22              issuer: "https://accounts.google.com"
23              rekor:
24                url: https://rekor.sigstore.dev

Keyless signing

To sign images using the keyless flow, use the following cosign command:

1COSIGN_EXPERIMENTAL=1 cosign sign ghcr.io/kyverno/test-verify-image:signed-keyless

This command generates ephemeral keys and launches a webpage to confirm an OIDC identity using providers like GitHub, Microsoft, or Google. The subject and issuer used in the policy must match the identity information provided during signing.

Keyless signing with GitHub Workflows

GitHub supports OpenID Connect (OIDC) tokens for workflow identities that eliminates the need for managing hard-coded secrets. A GitHub OIDC token can be used for keyless signing. In this case, the subject in the ephemeral certificate provides the identity of the workflow that executes the image signing tasks.

Since GitHub workflows can be reused in other workflows, it is important to verify the identity of both the executing workflow and the actual workflow used for signing. This can be done using attributes stored in X.509 certificate extensions.

The policy rule fragment checks for the subject and issuer from the certificate. The rule also checks for additional extensions registered using Fulcio Object IDs.

 1attestors:
 2- entries:
 3  - keyless:
 4      subject: "https://github.com/{{ORGANIZATION}}/{{REPOSITORY}}/.github/workflows/{{WORKFLOW}}@refs/tags/*"
 5      issuer: "https://token.actions.githubusercontent.com"
 6      additionalExtensions:
 7        githubWorkflowTrigger: push
 8        githubWorkflowSha: {{WORKFLOW_COMMIT_SHA}}
 9        githubWorkflowName: {{WORKFLOW_NAME}}
10        githubWorkflowRepository: {{WORKFLOW_ORGANIZATION}}/{{WORKFLOW_REPOSITORY}}
11      rekor:
12        url: https://rekor.sigstore.dev

Using a Key Management Service (KMS)

Kyverno and Cosign support using Key Management Services (KMS) such as AWS, GCP, Azure, and HashiCorp Vault. This integration allows referencing public and private keys using a URI syntax, instead of embedding the key directly in the policy.

The supported formats include:

azurekms://[VAULT_NAME][VAULT_URI]/[KEY]
awskms://[ENDPOINT]/[ID/ALIAS/ARN]
gcpkms://projects/[PROJECT]/locations/global/keyRings/[KEYRING]/cryptoKeys/[KEY]
hashivault://[KEY]

Refer to https://docs.sigstore.dev/cosign/kms_support for additional details.

Enabling IRSA to access AWS KMS

When running Kyverno in a AWS EKS cluster, you can use IAM Roles for Service Accounts (IRSA) to grant the Kyverno ServiceAccount permission to retrieve the public key(s) it needs from AWS KMS.

Once IRSA is enabled, the Kyverno ServiceAccount will have a new annotation with the IAM role it can assume, and the Kyverno Pod will assume this IAM role through the cluster’s OIDC provider. To understand how IRSA works internally, see links below:

Sample steps to enable IRSA for Kyverno using eksctl (see links above if you prefer to use AWS CLI instead):

Associate IAM OIDC provider

1eksctl utils associate-iam-oidc-provider --cluster <cluster-name> --approve

Create IAM policy

 1{
 2    "Version": "2012-10-17",
 3    "Statement": [
 4        {
 5            "Effect": "Allow",
 6            "Action": [
 7                "kms:GetPublicKey",
 8                "kms:DescribeKey"
 9            ],
10            "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
11        }
12    ]
13}

Create IAM role and annotate the Kyverno ServiceAccount with it

1eksctl create iamserviceaccount \
2    --name kyverno \
3    --namespace kyverno \
4    --cluster <cluster-name> \
5    --attach-policy-arn "arn:aws:iam::<account-id>:policy/<iam-policy>" \
6    --approve \
7    --override-existing-serviceaccounts

Note

Kyverno needs to know the AWS region for the KMS store in use. To provide this information, the environment variables AWS_DEFAULT_REGION and AWS_REGION need to be set in the Kyverno Deployment.

Verifying Image Annotations

Cosign has the ability to add annotations when signing an image, and Kyverno can be used to verify them. For example, this policy checks for the annotation of sig: original.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image
 5spec:
 6  validationFailureAction: Enforce
 7  background: false
 8  webhookTimeoutSeconds: 30
 9  failurePolicy: Fail
10  rules:
11    - name: check-image
12      match:
13        any:
14        - resources:
15            kinds:
16              - Pod
17      verifyImages:
18      - imageReferences: 
19        - ghcr.io/myorg/myimage*
20        attestors:
21        - entries:
22          - keys: 
23              publicKeys: |-
24                -----BEGIN PUBLIC KEY-----
25                MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEvy6wqHIx5JTxdDcbFIkb6boaxBBw
26                FZmwwzag3ZrsfOLT+r5DOx2LSyoef+eTda/QOcooUEZo7r4HpNbFH/y7Eg==
27                -----END PUBLIC KEY-----                
28            annotations:
29              sig: original

Using private registries

Private registries are defined as those requiring authentication in order to pull images. A private registry may also present leaf certificates issued by internal certificate authorities. To use a private registry, depending on which of these needs apply to you, check the following sections for instructions on how to configure Kyverno appropriately.

Authentication

In order for Kyverno to authenticate against a registry (private or otherwise), you must first create an imagePullSecret in the Kyverno Namespace and specify the Secret name as an argument to the Kyverno Deployment.

Configure the imagePullSecret:

1kubectl create secret docker-registry regcred \
2--docker-server=<server_address> \
3--docker-username=<username> \
4--docker-password=<password> \
5--docker-email=<email> \
6-n kyverno

Update the Kyverno Deployment to add the --imagePullSecrets=regcred argument:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  labels:
 5    app.kubernetes.io/component: kyverno
 6   ...
 7spec:
 8  replicas: 1
 9  selector:
10    matchLabels:
11      app: kyverno
12      app.kubernetes.io/name: kyverno
13  template:
14    spec:
15      containers:
16      - args:
17        ...
18        - --webhooktimeout=15
19        - --imagePullSecrets=regcred

Currently, only a single value for the --imagePullSecrets container flag is supported due to an upstream bug.

Trust

Kyverno does not by default have the same chain of trust as the underlying Kubernetes Nodes nor is it able to access them due to security concerns. Because the Nodes in your cluster can pull an image from a private registry (even if no authentication is required) does not mean Kyverno can. Kyverno ships with trust for the most common third-party certificate authorities and has no knowledge of internal PKI which may be in use by your private registry. Without the chain of trust established, Kyverno will not be able to fetch image metadata, signatures, or other OCI artifacts from a registry. Perform the following steps to present the necessary root certificates to Kyverno to establish trust.

There are two potential ways to have Kyverno trust your private registry. The first allows replacing all the certificates Kyverno trusts by default with only those needed by your internal registry. This has the benefit of being a simpler process at the cost of Kyverno losing trust for any public registries such as Docker Hub, Amazon ECR, GitHub Container Registry, etc. The second involves providing Kyverno with the same trust as your Nodes. Often times this trust includes the aforementioned public certificate authorities but in other cases may not. This first step involves the latter process.

Update your internal ca-certificates bundle by adding your private certificate authorities root certificate to the bundle and regenerating it. Many Linux distributions have slightly different processes, but it is documented here for Ubuntu as an example. If this process has already been done and your Nodes are using this, simply copy the contents out and proceed to the next step.

Create a ConfigMap in your cluster, preferably in the Kyverno Namespace, which has these contents stored as a multi-line value. It should look something like the below.

 1apiVersion: v1
 2kind: ConfigMap
 3metadata:
 4  name: kyverno-certs
 5  namespace: kyverno
 6data:
 7  ca-certificates: |
 8    -----BEGIN CERTIFICATE-----
 9    MIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE
10    AwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw
11    CQYDVQQGEwJFUzAeFw0xMTA1MDUwOTM3MzdaFw0zMDEyMzEwOTM3MzdaMEIxEjAQ
12    BgNVBAMMCUFDQ1ZSQUlaMTEQMA4GA1UECwwHUEtJQUNDVjENMAsGA1UECgwEQUND        
13<snip>
14    -----BEGIN CERTIFICATE-----
15    MIIBbzCCARWgAwIBAgIQK0Z1j0Q96/LIo4tNHxsPUDAKBggqhkjOPQQDAjAWMRQw
16    EgYDVQQDEwtab2xsZXJMYWJDQTAeFw0yMjA1MTgwODI2NTBaFw0zMjA1MTUwODI2
17    NTBaMBYxFDASBgNVBAMTC1pvbGxlckxhYkNBMFkwEwYHKoZIzj0CAQYIKoZIzj0D
18    AQcDQgAEJxGhyW26O77E7fqFcbzljYzlLq/G7yANNwerWnWUKlW9gcrcPqZwwrTX
19    yaJZpdCWTObvbOyaOxq5NsytC/ubLKNFMEMwDgYDVR0PAQH/BAQDAgEGMBIGA1Ud
20    EwEB/wQIMAYBAf8CAQEwHQYDVR0OBBYEFDoT1GEM8NYfxSKBkSzg4rpY+xdUMAoG
21    CCqGSM49BAMCA0gAMEUCIQDDLWFn/XJPqpNGXcyjlSJFxlQUJ5Cu/+nDvtbTeUGA
22    NAIgMsVwBafMtmLQFlfvZsE95UYoYUV4ayH+OLTTQaDQOPY=
23    -----END CERTIFICATE-----

Modify the Kyverno Deployment so it mounts this ConfigMap and overwrites the internal bundle provided by default. Refer to step one for the full ramifications of this act, especially if you have opted only to provide Kyverno with the certificate(s) of your internal root CA. An example snippet of the modified Deployment is shown below.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: kyverno
 5  namespace: kyverno
 6spec:
 7  template:
 8    spec:
 9      containers:
10      - image: ghcr.io/kyverno/kyverno:v1.9.0
11        name: kyverno
12        volumeMounts:
13        - mountPath: /.sigstore
14          name: sigstore
15        - name: ca-certificates
16          mountPath: /etc/ssl/certs/ca-certificates.crt
17          subPath: ca-certificates.crt
18<snip>
19      volumes:
20      - name: sigstore
21      - name: ca-certificates
22        configMap:
23          name: kyverno-certs
24          items:
25          - key: ca-certificates
26            path: ca-certificates.crt

Using a signature repository

To use a separate registry to store signatures use the COSIGN_REPOSITORY environment variable when signing the image. Then in the Kyverno policy rule, specify the repository for each image:

 1...
 2verifyImages:
 3- imageReferences:
 4  - ghcr.io/kyverno/test-verify-image*
 5  repository: "registry.io/signatures"
 6  attestors:
 7  - entries:
 8    - keys:
 9        publicKeys: |-
10          -----BEGIN PUBLIC KEY-----
11          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
12          5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
13          -----END PUBLIC KEY-----          
14...

Verifying images in Custom Resources

In addition to Kubernetes Pods, custom resources such as Tekton Tasks, Argo Workflow Steps, and others may also reference container images. In other cases, rather than an image, an OCI artifact like a Tekton Pipeline bundle may be signed. Kyverno supports verification for images and OCI Artifacts in custom resources by allowing the declaration of an imageExtractor, which specifies the location of the image or artifact in the custom resource.

Here is an example of a policy that verifies that Tekton task steps are signed using a private key that matches the specified public key:

 1
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: signed-task-image
 6spec:
 7  validationFailureAction: Enforce
 8  rules:
 9  - name: check-signature
10    match:
11      any:
12      - resources:
13          kinds:
14          - tekton.dev/v1beta1/TaskRun.status
15    imageExtractors:
16      TaskRun:
17        - name: "taskrunstatus"
18          path: "/status/taskSpec/steps/*"
19          value: "image"
20          key: "name"
21    verifyImages:
22    - imageReferences:
23      - "*"
24      required: false
25      attestors:
26      - entries:
27        - keys: 
28            publicKeys: |-
29              -----BEGIN PUBLIC KEY-----
30              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEahmSvGFmxMJABilV1usgsw6ImcQ/
31              gDaxw57Sq+uNGHW8Q3zUSx46PuRqdTI+4qE3Ng2oFZgLMpFN/qMrP0MQQg==
32              -----END PUBLIC KEY-----

This policy rule checks that Tekton pipeline bundles are signed with a private key that matches the specified public key:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: signed-pipeline-bundle
 5spec:
 6  validationFailureAction: Enforce
 7  rules:
 8  - name: check-signature
 9    match:
10      any:
11      - resources:
12          kinds:
13          - PipelineRun
14    imageExtractors:
15      PipelineRun:
16        - name: "pipelineruns"
17          path: /spec/pipelineRef
18          value: "bundle"
19          key: "name"
20    verifyImages:
21    - imageReferences:
22      - "*"
23      attestors:
24      - entries:
25        - keys: 
26            publicKeys: |-
27              -----BEGIN PUBLIC KEY-----
28              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEahmSvGFmxMJABilV1usgsw6ImcQ/
29              gDaxw57Sq+uNGHW8Q3zUSx46PuRqdTI+4qE3Ng2oFZgLMpFN/qMrP0MQQg==
30              -----END PUBLIC KEY-----

For Custom Resources which reference container images in a non-standard way, an optional jmesPath field may be used to apply a filter to transform the value of the extracted field. For example, in the case of KubeVirt’s DataVolume custom resource, the fielding referencing the image needing verification is located at spec.source.registry.url as seen below.

 1apiVersion: cdi.kubevirt.io/v1beta1
 2kind: DataVolume
 3metadata:
 4  name: registry-image-datavolume
 5spec:
 6  source:
 7    registry:
 8      url: "docker://kubevirt/fedora-cloud-registry-disk-demo"
 9  pvc:
10    accessModes:
11      - ReadWriteOnce
12    resources:
13      requests:
14        storage: 5Gi

The value of the field contains a prefix of docker:// which must be removed first. Applying a JMESPath expression in an extractor along with a Kyverno custom filter such as trim_prefix() can be used to provide the container image for Kyverno to verify.

1imageExtractors:
2  DataVolume:
3    - path: /spec/source/registry/url
4      jmesPath: "trim_prefix(@, 'docker://')"

Special Variables

A pre-defined, reserved special variable named image is available for use only in verifyImages rules. The following fields are available under the image object and may be used in a rule to reference the named fields.

reference
referenceWithTag
registry
path
name
tag
digest

Offline Registries

The policy-level setting failurePolicy when set to Ignore additionally means that failing calls to image registries will be ignored. This allows for Pods to not be blocked if the registry is offline, useful in situations where images already exist on the nodes.

Known Issues

Check the Kyverno GitHub repo for a list of pending issues for this feature.

Last modified May 30, 2023 at 11:31 AM PST: Add scale testing results (#877) (8ff14e5)