Scanning vulnerabilities during continuous deployment of Managed Service for Kubernetes applications using GitLab

You can scan Docker images for vulnerabilities in Yandex Container Registry when continuously deploying Yandex Managed Service for Kubernetes applications via GitLab.

To do this, use Continuous Integration (CI) to create a special script in GitLab that will run after each commit:

1. Building an application into a Docker image and pushing the image to Container Registry.
2. Scanning Docker images in Container Registry for vulnerabilities.
3. Deploying an application from a Docker image in a Managed Service for Kubernetes cluster using the Yandex Cloud tools.

To set up the vulnerability scanner:

1. Create a GitLab instance.
2. Configure GitLab.
3. Create a test application.
4. Create a GitLab Runner.
5. Configure the CI script.
6. Check the result.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for disks and continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for storing the created Docker images and for the vulnerability scanner (see Container Registry pricing).
• Fee for using the Managed Service for Kubernetes master (see Managed Service for Kubernetes pricing).
• Fee for using public IP addresses (see Yandex Virtual Private Cloud pricing).

Getting startedGetting started

Set up your infrastructureSet up your infrastructure

Additionally, assign the container-registry.images.scanner role to the service account for the nodes. This role allows scanning Docker images for vulnerabilities.

Install additional dependenciesInstall additional dependencies

Install the following items in the local environment:

• jq JSON stream processor
• kubectl command-line tool. Configure it to work with the created Managed Service for Kubernetes cluster.

Create a GitLab instanceCreate a GitLab instance

Create either a Managed Service for GitLab instance or a VM with a GitLab image in the same cloud network as the Managed Service for Kubernetes cluster.

Configure GitLabConfigure GitLab

To configure GitLab and enable Continuous Integration (CI), create a new project and enter the CI authorization parameters:

Create a test applicationCreate a test application

Create a test application that can be deployed in a Yandex Managed Service for Kubernetes cluster:

1. Add a Dockerfile to the project:

   1. Log in to GitLab.

   2. Open the GitLab project.

   3. Click in the repository navigation bar and select New file from the drop-down menu.

   4. Name the file as Dockerfile and add the following code to it:

      FROM alpine:3.10
      CMD echo "Hello"
      

   5. Add a comment to the commit in the Commit message field: Dockerfile for test application.

   6. Click Commit changes.

2. Add the manifest for creating Managed Service for Kubernetes cluster resources to the project:

   1. Open the GitLab project.

   2. Click in the repository navigation bar and select New file from the drop-down menu.

   3. Name the file as k8s.yaml:

      k8s.yaml

      apiVersion: v1
      kind: Namespace
      metadata:
        name: hello-world
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: hello-world-deployment
        namespace: hello-world
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: hello
        template:
          metadata:
            namespace: hello-world
            labels:
              app: hello
          spec:
            containers:
              - name: hello-world
                image: __VERSION__
                imagePullPolicy: Always
      

   4. Add a comment to the commit in the Commit message field: Docker image deployment config.

   5. Click Commit changes.

Create a GitLab RunnerCreate a GitLab Runner

To run build tasks in the Yandex Managed Service for Kubernetes cluster, create a GitLab Runner. To do this, install GitLab Runner by following this guide.

Once it is installed, you can run automated builds inside your Managed Service for Kubernetes cluster.

For more information about installing and running GitLab Runner, see the GitLab documentation.

Set up Kubernetes authentication in GitLabSet up Kubernetes authentication in GitLab

You can set up authentication in GitLab using a Kubernetes service account token or the GitLab Agent application:

Configure the CI scriptConfigure the CI script

1. Create the GitLab environment variables:

   1. In GitLab, navigate to Settings in the left-hand panel and select CI/CD from the drop-down list.

   2. Click Expand next to Variables.

   3. Add the following environment variables depending on the Managed Service for Kubernetes authentication method in GitLab:

      Service account token

      GitLab Agent

      • KUBE_URL: Managed Service for Kubernetes master address. You can retrieve it using the following command:

        yc managed-kubernetes cluster get <cluster_name_or_ID> --format=json \
           | jq -r .master.endpoints.external_v4_endpoint
        

      • KUBE_TOKEN: Token that GitLab will use to apply the configuration. Use the token obtained earlier.
      • CI_REGISTRY: Address of the previously created registry in cr.yandex/<registry_ID> format.
      • CI_REGISTRY_KEY: Key that GitLab will use to access the registry. Copy the contents of the previously obtained key.json static key file to access the registry.

      To add a variable:

      • Click Add variable.
      • In the window that opens, specify the variable name in the Key field and its value in the Value field.
      • Click Add variable.

2. Create the CI script configuration file:

   1. Open the gitlab-test project.

   2. Click in the repository navigation bar and select New file from the drop-down menu.

   3. Name your file .gitlab-ci.yml. Add the steps to build and push a Docker image, scan it for vulnerabilities, and update the application configuration in the Managed Service for Kubernetes cluster. The file structure depends on the Kubernetes authentication method in GitLab:

      Service account token

      GitLab Agent

      .gitlab-ci.yml

      stages:
        - build
        - test
        - deploy
      
      build_docker_image:
        stage: build
        variables:
          DOCKER_CUSTOM_SUBFOLDER: "" # Specify a custom path (if any) to your folder with docker files.
        image:
          name: gcr.io/kaniko-project/executor:debug
          entrypoint: [""]
        script:
          - mkdir -p /kaniko/.docker
          - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
          - >-
            /kaniko/executor
            --context "${CI_PROJECT_DIR}"/"${DOCKER_CUSTOM_SUBFOLDER}"
            --dockerfile "${CI_PROJECT_DIR}/"${DOCKER_CUSTOM_SUBFOLDER}"/Dockerfile"
            --destination "${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}"
      
      container_scanning_free_yc:
        image: 
          name: pindar/jq
          entrypoint: [""]
        stage: test
        artifacts:
          when: always
          paths:
            - gl-container-scanning-report-yc.json
        script:
          - export CI_COMMIT_SHA=${CI_COMMIT_SHA}
          # Install YC CLI.
          - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/
          # Start scanning.
          - echo "Scanning image $IMAGE_NAME ${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}..."
          - export IMAGE_ID=$(yc container image list --registry-id (${CI_REGISTRY} | cut -d/ -f2) --format=json | jq -r --arg CI_COMMIT_SHA $CI_COMMIT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHA) | .id ')
          # Make a report.
          - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json)
          - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0')
          - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0')
          - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id')
          - echo "Scan results:"
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")'
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json
          # Check the result.
          - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulns and $HIGH_VULN high vulns" && exit 1 || echo "image has no high or crit vulns" exit 0
      
      deploy:
        image: gcr.io/cloud-builders/kubectl:latest
        stage: deploy
        script:
          - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
          - kubectl config set-credentials admin --token="$KUBE_TOKEN"
          - kubectl config set-context default --cluster=k8s --user=admin
          - kubectl config use-context default
          - sed -i "s,__VERSION__,${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}," k8s.yaml
          - kubectl apply -f k8s.yaml
      

      .gitlab-ci.yml

      stages:
        - build
        - test
        - deploy
      
      build_docker_image:
        stage: build
        variables:
          DOCKER_CUSTOM_SUBFOLDER: "" # Specify a custom path (if any) to your folder with docker files.
        image:
          name: gcr.io/kaniko-project/executor:debug
          entrypoint: [""]
        script:
          - mkdir -p /kaniko/.docker
          - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
          - >-
            /kaniko/executor
            --context "${CI_PROJECT_DIR}"/"${DOCKER_CUSTOM_SUBFOLDER}"
            --dockerfile "${CI_PROJECT_DIR}/"${DOCKER_CUSTOM_SUBFOLDER}"/Dockerfile"
            --destination "${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}"
      
      container_scanning_free_yc:
        image: 
          name: pindar/jq
          entrypoint: [""]
        stage: test
        artifacts:
          when: always
          paths:
            - gl-container-scanning-report-yc.json
        script:
          - export CI_COMMIT_SHA=${CI_COMMIT_SHA}
          # Install YC CLI.
          - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/
          # Start scanning.
          - echo "Scanning image $IMAGE_NAME ${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}..."
          - export IMAGE_ID=$(yc container image list --registry-id (${CI_REGISTRY} | cut -d/ -f2) --format=json | jq -r --arg CI_COMMIT_SHA $CI_COMMIT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHA) | .id ')
          # Make a report.
          - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json)
          - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0')
          - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0')
          - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id')
          - echo "Scan results:"
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")'
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json
          # Check the result.
          - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulns and $HIGH_VULN high vulns" && exit 1 || echo "image has no high or crit vulns" exit 0
      
      deploy:
        image: bitnami/kubectl:latest
        stage: deploy
        script:
          - kubectl config use-context ${CI_PROJECT_PATH}:<GitLab_Agent_name>
          - cat k8s.yaml | sed -e "s,__VERSION__,${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}," | kubectl apply -f -
      

      Replace <GitLab_Agent_name> with the name of the agent in Managed Service for GitLab.

   4. Add a comment in the Commit message field: CI scripts.

   5. Click Commit changes.

   .gitlab-ci.yml describes the following CI script steps:

   • Building a Docker image using Dockerfile and pushing the image to Container Registry.
   • Scanning the Docker image for vulnerabilities in Container Registry.
   • Setting up an environment to work with Kubernetes and applying the k8s.yaml configuration to Managed Service for Kubernetes clusters. This way, the application is deployed on the previously created Managed Service for Kubernetes cluster.

Check the resultCheck the result

After you save the .gitlab-ci.yml configuration file, the build script will start. To check its results, select Build on the left-hand panel in the gitlab-test project, and then Pipelines from the drop-down menu. Vulnerability scanning is performed at the second stage (test).

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the Managed Service for GitLab instance or the created VM with the GitLab image.
2. Delete all Docker images from the Container Registry registry.

Delete the other resources depending on how they were created: