Storing Docker images created in Yandex Managed Service for GitLab projects

Container Registry is integrated into GitLab. It enables you to store Docker images on a per-project basis in GitLab.

You can use Yandex Container Registry instead of GitLab Container Registry. This service allows you to store Docker images in the cloud and share them across Yandex Cloud managed services, such as Yandex Managed Service for Kubernetes or Yandex Managed Service for GitLab.

Using Yandex Container Registry to store images from GitLab projects has several advantages:

• GitLab Container Registry stores images and tags on the GitLab instance disk. When the instance runs out of disk space, it returns HTTP 500 errors and becomes unavailable. You can restore the instance only by contacting support.

  Yandex Container Registry stores images and tags in registries, each with its own quota. Therefore, storing large numbers of Docker images and tags does not consume the instance disk space.

• Images stored in Yandex Container Registry remain available even if Managed Service for GitLab becomes unavailable.

• Yandex Container Registry supports the Docker image vulnerability scanner. Use it to detect vulnerabilities and fix them before deploying your application.

To set up storage of Managed Service for GitLab Docker images in Yandex Container Registry:

1. Create a GitLab instance.
2. Configure GitLab.
3. Create a test application.
4. Create a GitLab Runner.
5. Create GitLab environment variables.
6. Create a CI pipeline configuration file.
7. Check the result.
8. Enable a Docker image lifecycle policy.
9. Optionally, scan your Docker images for vulnerabilities.

If you have already configured your Yandex Managed Service for GitLab instance for Continuous Integration (CI), make sure to set up your infrastructure for storing Docker images. Then, proceed with the setup by creating environment variables.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

Infrastructure support costs include fees for the following resources:

• Disks and continuously running VMs (see Yandex Compute Cloud pricing).
• Dynamic public IP address assigned (see Yandex Virtual Private Cloud pricing).
• Storage of the Docker images you created and vulnerability scanner, if enabled (see Container Registry pricing).
• Managed Service for Kubernetes master (see Managed Service for Kubernetes pricing).

Set up your infrastructureSet up your infrastructure

Create a GitLab instanceCreate a GitLab instance

Create either a Managed Service for GitLab instance or a VM with a GitLab image in the same cloud network as the Managed Service for Kubernetes cluster.

Configure GitLabConfigure GitLab

To configure GitLab and get ready for continuous integration (CI), create a new project and enter the CI authentication data:

Create a test applicationCreate a test application

Create a test application that can be deployed in a Managed Service for Kubernetes cluster. To do this, add a Dockerfile to your project:

1. Log in to GitLab.

2. Open your GitLab project.

3. Click in the repository navigation bar and select New file from the drop-down menu.

4. Name the file Dockerfile and add the following code to it:

   FROM alpine:3.10
   CMD echo "Hello"
   

5. Add a comment in the Commit message field: Dockerfile for a test application.

6. Click Commit changes.

Create a GitLab RunnerCreate a GitLab Runner

To run build tasks in your Yandex Managed Service for Kubernetes cluster, create a GitLab Runner.To do this, install GitLab Runner by following this guide.

Once it is installed, you can run automated builds within your Managed Service for Kubernetes cluster.

For more information about installing and running GitLab Runner, see this GitLab article.

Create GitLab environment variablesCreate GitLab environment variables

To allow Managed Service for GitLab to save Docker images and their tags to Yandex Container Registry, create GitLab environment variables:

1. If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

2. Create an authorized key for the account-for-container-registry service account you created earlier:

   yc iam key create --service-account-name account-for-container-registry -o key.json
   

   The key is saved in the key.json file in the current working directory.

3. Open your project in GitLab.

4. Go to Settings in the left-hand panel and select CI/CD from the drop-down list.

5. Expand the Variables section.

6. Add environment variables with the protection option disabled:

   Variable	Value
   CI_REGISTRY	cr.yandex/<registry_ID>. Specify the ID of the registry you created earlier in Yandex Container Registry.
   CI_REGISTRY_KEY	key.json file contents

   To add a variable:

   1. Click Add variable.

   2. For the CI_REGISTRY_KEY variable, select the Visible option in the Visibility field.

      Warning

      The key.json file contains spaces and line breaks which cannot be masked in GitLab. However, in the provided CI pipeline example, the CI_REGISTRY_KEY variable is not displayed explicitly, so you do not need to mask it.

   3. In the window that opens, specify a variable name in the Key field and its value in the Value field.

   4. Disable Protect variable.

   5. Click Add variable.

Create a CI pipeline configuration fileCreate a CI pipeline configuration file

To build images from a Dockerfile without Docker, use kaniko.

To publish Docker images from your GitLab project to Yandex Container Registry, create a CI pipeline configuration:

1. Open the gitlab-test project.

2. Click in the repository navigation bar and select New file from the drop-down menu.

3. Name your file .gitlab-ci.yml. Add the steps for building the Docker image and pushing it to Yandex Container Registry:

   .gitlab-ci.yml

   build:
      stage: build
      # Using `kaniko` to create a container inside another container for enhanced security.
      image:
         name: gcr.io/kaniko-project/executor:debug
         entrypoint: [""]
      script:
         - mkdir -p /kaniko/.docker
         # Pushing the container image to the registry. The image is tagged with the commit hash.
         - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
         - >-
            /kaniko/executor
            --context "${CI_PROJECT_DIR}"
            --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
            --destination "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}"
   

   This file includes the following variables:

   • CI_REGISTRY and CI_REGISTRY_KEY: Added to GitLab in the previous step.
   • CI_PROJECT_DIR, CI_PROJECT_PATH, and CI_COMMIT_SHORT_SHA: Preset in GitLab.

4. Add a comment in the Commit message field: Create a CI pipeline.

5. Click Commit changes.

Check the resultCheck the result

Each commit triggers a build pipeline. To verify that it completed successfully:

1. Select Build in the left-hand panel in the gitlab-test project, and then select Pipelines from the drop-down menu.

2. Make sure the build stage has the passed status. This means that the CI pipeline completed successfully.

3. Go to the management console and open your Yandex Container Registry.

   If the CI pipeline completes successfully, your registry will contain a new repository. New Docker images from the GitLab project will be added to this repository at each commit.

Enable a Docker image lifecycle policyEnable a Docker image lifecycle policy

To avoid storing outdated Docker images and their tags, configure a Docker image lifecycle policy. It applies to images in your Container Registry repository and enables timely cleanup of the repository. This way, you do not pay extra for storing outdated images.

To create a policy, follow this guide.

When using a policy:

• External Container Registry and Docker image lifecycle policy affect CI pipeline performance.

• The policy has a limit on the maximum number of images that can be checked per policy run. If the number of images in your Container Registry repository exceeds this limit, run the policy several times. This way, all your images will be checked.

Scan your Docker images for vulnerabilitiesScan your Docker images for vulnerabilities

To detect vulnerabilities in your Docker images, you can additionally enable a vulnerability scanner in Yandex Container Registry. The scanner checks the versions of packages installed in your images against CVE vulnerability databases.

To enable scanning, update the CI pipeline in your GitLab project:

1. Open the gitlab-test project.

2. Open the .gitlab-ci.yml file.

3. Add the steps to scan the Docker image for vulnerabilities:

   .gitlab-ci.yml

   stages:
      - build
      - test
   
   <build_block_previously_added_to_file>
   
   container_scanning_free_yc:
      stage: test
      # Using jq to search for ID and write logs.
      image: 
         name: pindar/jq
         entrypoint: [""]
      artifacts:
         when: always
         paths:
            - gl-container-scanning-report-yc.json
      variables:
         # Specify the ID of the registry you previously created.
         CI_REGISTRY_ID: "<registry_ID>"
      script:
         - export CI_COMMIT_SHORT_SHA=${CI_COMMIT_SHORT_SHA}
         # Installing the Yandex Cloud CLI.
         - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/
         # Start of scanning.
         - echo "Scanning image ${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}..."
         - export IMAGE_ID=$(yc container image list --registry-id $CI_REGISTRY_ID --format=json | jq -r --arg CI_COMMIT_SHORT_SHA $CI_COMMIT_SHORT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHORT_SHA) | .id ')
         # Logging.
         - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json)
         - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0')
         - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0')
         - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id')
         - echo "Scan results:"
         - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")'
         - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json
         # Checking the result.
         - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulnerabilities and $HIGH_VULN high vulnerabilities" && exit 1 || echo "image has no high or critical vulnerabilities" exit 0
   

4. Add a comment in the Commit message field: Turn on a vulnerability scanner.

5. Click Commit changes. After that, the updated pipeline will run.

To make sure the image scan completed successfully:

1. Select Build in the left-hand panel in the gitlab-test project, and then select Pipelines from the drop-down menu.

2. Make sure the build and test stages have the passed status. This means that the CI pipeline completed successfully.

3. Go to the management console and open your Yandex Container Registry.

4. Open the repository with Docker images from the GitLab project.

5. Go to the directory named after the GitLab project.

6. Make sure the Scan status column shows Ready.

7. In the Date of last scan column, click the link with the scan time.

   This will open the scan result page. Any vulnerabilities found in the image will be displayed there.

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the Managed Service for GitLab instance or the GitLab VM.
2. Delete all Docker images from the Container Registry registry.

Delete the other resources depending on how you created them: