Scanning vulnerabilities during continuous deployment of Managed Service for Kubernetes applications using GitLab

You can scan Docker images for vulnerabilities in Yandex Container Registry when continuously deploying Yandex Managed Service for Kubernetes applications via GitLab.

To do this, use Continuous Integration (CI) to create a special script in GitLab that will run after each commit:

1. Building an application into a Docker image and pushing the image to Container Registry.
2. Scanning Docker images in Container Registry for vulnerabilities.
3. Deploying an application from a Docker image in a Managed Service for Kubernetes cluster using the Yandex Cloud tools.

To set up the vulnerability scanner:

1. Create a GitLab instance.
2. Configure GitLab.
3. Create a test application.
4. Create a GitLab Runner.
5. Configure the CI script.
6. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructurePrepare the infrastructure

Install additional dependenciesInstall additional dependencies

Install the following items in the local environment:

• jq JSON stream processor
• kubectl command-line tool. Configure it to work with the created Managed Service for Kubernetes cluster.

Create a GitLab instanceCreate a GitLab instance

Create a Managed Service for GitLab instance or a VM with a GitLab image on the same cloud network as the Managed Service for Kubernetes cluster.

Configure GitLabConfigure GitLab

To configure GitLab and enable Continuous Integration (CI), create a new project and enter the CI authorization parameters:

Create a test applicationCreate a test application

Create a test application that can be deployed in a Yandex Managed Service for Kubernetes cluster:

1. Add a Dockerfile to the project:

   1. Log in to GitLab.

   2. Open the GitLab project.

   3. Click in the repository navigation bar and select New file from the drop-down menu.

   4. Name the file as Dockerfile and add the following code to it:

      FROM alpine:3.10
      CMD echo "Hello"
      

   5. Add a comment to the commit in the Commit message field: Dockerfile for test application.

   6. Click Commit changes.

2. Add the manifest for creating Managed Service for Kubernetes cluster resources to the project:

   1. Open the GitLab project.

   2. Click in the repository navigation bar and select New file from the drop-down menu.

   3. Name the file as k8s.yaml:

      k8s.yaml

      apiVersion: v1
      kind: Namespace
      metadata:
        name: hello-world
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: hello-world-deployment
        namespace: hello-world
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: hello
        template:
          metadata:
            namespace: hello-world
            labels:
              app: hello
          spec:
            containers:
              - name: hello-world
                image: __VERSION__
                imagePullPolicy: Always
      

   4. Add a comment to the commit in the Commit message field: Docker image deployment config.

   5. Click Commit changes.

Create a GitLab RunnerCreate a GitLab Runner

To run build tasks in the Yandex Managed Service for Kubernetes cluster, create a GitLab Runner. To do this, install the GitLab Runner application by following this guide.

Once it is installed, you can run automated builds inside your Managed Service for Kubernetes cluster.

For more information about installing and running GitLab Runner, see the GitLab documentation.

Set up Kubernetes authentication in GitLabSet up Kubernetes authentication in GitLab

You can set up authentication in GitLab using a Kubernetes service account token or the GitLab Agent application:

Configure the CI scriptConfigure the CI script

1. Create the GitLab environment variables:

   1. In GitLab, go to Settings in the left-hand panel and select CI/CD from the drop-down list.

   2. Click Expand next to Variables.

   3. Add the following environment variables depending on the Managed Service for Kubernetes authentication method in GitLab:

      Service account token

      GitLab Agent

      • KUBE_URL: Managed Service for Kubernetes master address. You can retrieve it using the following command:

        yc managed-kubernetes cluster get <cluster_name_or_ID> --format=json \
           | jq -r .master.endpoints.external_v4_endpoint
        

      • KUBE_TOKEN: Token that will use GitLab to apply the configuration. Use the token obtained earlier.
      • CI_REGISTRY: Address of the previously created registry in cr.yandex/<registry_ID> format.
      • CI_REGISTRY_KEY: Key that GitLab will use to access the registry. Copy the contents of the previously obtained key.json static key file to access the registry.

      To add a variable:

      • Click Add variable.
      • In the window that opens, enter the variable name in the Key field and the value in the Value field.
      • Click Add variable.

2. Create the CI script configuration file:

   1. Open the gitlab-test project.

   2. Click in the repository navigation bar and select New file from the drop-down menu.

   3. Name the file as .gitlab-ci.yml. Add the steps to build and push a Docker image, scan it for vulnerabilities, and update the application configuration in the Managed Service for Kubernetes cluster. The file structure depends on the Kubernetes authentication method in GitLab:

      Service account token

      GitLab Agent

      .gitlab-ci.yml

      stages:
        - build
        - test
        - deploy
      
      build_docker_image:
        stage: build
        variables:
          DOCKER_CUSTOM_SUBFOLDER: "" # Specify a custom path (if any) to your folder with docker files.
        image:
          name: gcr.io/kaniko-project/executor:debug
          entrypoint: [""]
        script:
          - mkdir -p /kaniko/.docker
          - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
          - >-
            /kaniko/executor
            --context "${CI_PROJECT_DIR}"/"${DOCKER_CUSTOM_SUBFOLDER}"
            --dockerfile "${CI_PROJECT_DIR}/"${DOCKER_CUSTOM_SUBFOLDER}"/Dockerfile"
            --destination "${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}"
      
      container_scanning_free_yc:
        image:
          name: pindar/jq
          entrypoint: [""]
        stage: test
        artifacts:
          when: always
          paths:
            - gl-container-scanning-report-yc.json
        script:
          - export CI_COMMIT_SHA=${CI_COMMIT_SHA}
          # Install YC CLI.
          - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/
          # Start scanning.
          - echo "Scanning image $IMAGE_NAME ${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}..."
          - export IMAGE_ID=$(yc container image list --registry-id (${CI_REGISTRY} | cut -d/ -f2) --format=json | jq -r --arg CI_COMMIT_SHA $CI_COMMIT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHA) | .id ')
          # Make a report.
          - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json)
          - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0')
          - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0')
          - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id')
          - echo "Scan results:"
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")'
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json
          # Check the result.
          - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulns and $HIGH_VULN high vulns" && exit 1 || echo "image has no high or crit vulns" exit 0
      
      deploy:
        image: gcr.io/cloud-builders/kubectl:latest
        stage: deploy
        script:
          - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
          - kubectl config set-credentials admin --token="$KUBE_TOKEN"
          - kubectl config set-context default --cluster=k8s --user=admin
          - kubectl config use-context default
          - sed -i "s,__VERSION__,${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}," k8s.yaml
          - kubectl apply -f k8s.yaml
      

      .gitlab-ci.yml

      stages:
        - build
        - test
        - deploy
      
      build_docker_image:
        stage: build
        variables:
          DOCKER_CUSTOM_SUBFOLDER: "" # Specify a custom path (if any) to your folder with docker files.
        image:
          name: gcr.io/kaniko-project/executor:debug
          entrypoint: [""]
        script:
          - mkdir -p /kaniko/.docker
          - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
          - >-
            /kaniko/executor
            --context "${CI_PROJECT_DIR}"/"${DOCKER_CUSTOM_SUBFOLDER}"
            --dockerfile "${CI_PROJECT_DIR}/"${DOCKER_CUSTOM_SUBFOLDER}"/Dockerfile"
            --destination "${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}"
      
      container_scanning_free_yc:
        image:
          name: pindar/jq
          entrypoint: [""]
        stage: test
        artifacts:
          when: always
          paths:
            - gl-container-scanning-report-yc.json
        script:
          - export CI_COMMIT_SHA=${CI_COMMIT_SHA}
          # Install YC CLI.
          - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/
          # Start scanning.
          - echo "Scanning image $IMAGE_NAME ${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}..."
          - export IMAGE_ID=$(yc container image list --registry-id (${CI_REGISTRY} | cut -d/ -f2) --format=json | jq -r --arg CI_COMMIT_SHA $CI_COMMIT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHA) | .id ')
          # Make a report.
          - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json)
          - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0')
          - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0')
          - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id')
          - echo "Scan results:"
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")'
          - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json
          # Check the result.
          - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulns and $HIGH_VULN high vulns" && exit 1 || echo "image has no high or crit vulns" exit 0
      
      deploy:
        image: bitnami/kubectl:latest
        stage: deploy
        script:
          - kubectl config use-context ${CI_PROJECT_PATH}:<GitLab_Agent_name>
          - cat k8s.yaml | sed -e "s,__VERSION__,${CI_REGISTRY}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}," | kubectl apply -f -
      

      Replace <GitLab_Agent_name> with the agent name in Managed Service for GitLab.

   4. Add a comment to the commit in the Commit message field: CI scripts.

   5. Click Commit changes.

   In the .gitlab-ci.yml file, the following steps of the CI script are described:

   • Building a Docker image using the Dockerfile and pushing the image to Container Registry.
   • Scanning the Docker image for vulnerabilities in Container Registry.
   • Setting up an environment to work with Kubernetes and applying k8s.yaml configurations to Managed Service for Kubernetes clusters. This way, the application is deployed on the previously created Managed Service for Kubernetes cluster.

Check the resultCheck the result

After you save the .gitlab-ci.yml configuration file, the build script will start. To check its results, select Build on the left-hand panel in the gitlab-test project, and then choose Pipelines from the drop-down menu. Vulnerability scanning is performed at the second stage (test).

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the Managed Service for GitLab instance or the created VM with the GitLab image.
2. Delete all Docker images from the Container Registry registry.

Delete the other resources, depending on the method used to create them: