Storing Docker images created in Yandex Managed Service for GitLab projects
- Getting started
- Create a GitLab instance
- Configure GitLab
- Create a test application
- Create a GitLab Runner
- Create GitLab environment variables
- Create the CI script configuration file
- Check the result
- Enable a Docker image lifecycle policy
- Scan your Docker images for vulnerabilities
- Delete the resources you created
Container Registry
You can use Yandex Container Registry instead of GitLab Container Registry. This service enables you to store Docker images in the cloud or distribute them across Yandex Cloud managed services, for example, Yandex Managed Service for Kubernetes or Yandex Managed Service for GitLab.
Storing images from GitLab projects in Yandex Container Registry has several benefits:
-
GitLab Container Registry stores images and tags on the GitLab instance disk. When you run out of disk space, you get the HTTP 500 error, and the instance becomes unavailable. You can recover the instance only by contacting tech support.
Yandex Container Registry stores images and tags in registries for which individual quotas are allocated. Because of this, accumulating Docker images and tags does not affect the space available on the instance disk.
-
The images are still available in Yandex Container Registry, even if Managed Service for GitLab is not.
-
Yandex Container Registry supports the Docker image vulnerability scanner. Use the scanner to detect vulnerabilities and fix them before deploying your application. You can learn more about security scans in the Yandex Cloud blog.
To set up storage of Managed Service for GitLab Docker images in Yandex Container Registry:
- Create a GitLab instance.
- Configure GitLab.
- Create a test application.
- Create a GitLab Runner.
- Create GitLab environment variables.
- Create the CI script configuration file.
- Check the result.
- Enable a Docker image lifecycle policy.
- (Optional) Scan your Docker images for vulnerabilities.
If you have already set up your Yandex Managed Service for GitLab instance for Continuous Integration (CI), check that you have prepared the infrastructure for Docker images. That done, begin the setup by creating environment variables.
If you no longer need the resources you created, delete them.
Note
By default, GitLab Container Registry is disabled when creating an Managed Service for GitLab instance.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
Infrastructure support costs include fees for the following resources:
- Disks and continuously running VMs (see Yandex Compute Cloud pricing).
- Using a dynamic public IP address (see Yandex Virtual Private Cloud pricing).
- Storing the Docker images you created and a vulnerability scanner, if activated (see Container Registry pricing).
- Using a Managed Service for Kubernetes master (see Managed Service for Kubernetes pricing).
Prepare the infrastructure
-
If you do not have a network yet, create one.
-
If you do not have any subnets, create them in the availability zones where your Yandex Managed Service for Kubernetes cluster and node group will be created.
-
Create a service account named
account-for-container-registry
with roles for the folder.editor
container-registry.images.pusher
container-registry.images.puller
-
Create a Managed Service for Kubernetes cluster with a zonal master and a node group. When creating a cluster, specify the service account you created previously.
-
Configure a security group for the Managed Service for Kubernetes cluster and Managed Service for GitLab instance.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the container-registry-and-gitlab.tf
configuration file to the same working directory.This file describes:
- Network.
- Subnet.
- Security group and rules required for running the Managed Service for GitLab instance and Yandex Managed Service for Kubernetes cluster.
- Managed Service for Kubernetes cluster with a zonal master.
- Node group for the cluster.
- Service account required to use the Managed Service for Kubernetes cluster and node group.
- Yandex Container Registry registry.
-
In the
container-registry-and-gitlab.tf
file, specify:- Cloud ID.
- Folder ID.
- Kubernetes version for the Managed Service for Kubernetes cluster and node groups.
-
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Create a GitLab instance
Create a Managed Service for GitLab instance or a VM with a GitLab image on the same cloud network as the Managed Service for Kubernetes cluster.
Create a Managed Service for GitLab instance by following this guide.
Launch GitLab on a VM with a public IP.
- On the folder page in the management console
, click Create resource and select Virtual machine instance. - In the Name field, enter the VM name as follows:
ci-tutorial-gitlab
. - Select an availability zone to place your VM in.
- Under Boot disk image, go to the Marketplace tab and click Show all Marketplace products. In the window that opens, select GitLab as your image and click Use.
- Under Computing resources, specify the following configuration:
- vCPU:
4
- Guaranteed vCPU performance:
100%
- RAM:
8 GB
- vCPU:
- Under Network settings:
-
Select a subnet to connect your VM to. If the required network or subnet is not listed, create it using the Create network and Create subnet buttons.
Warning
For the time being, Yandex Cloud technical restrictions do not allow selecting a subnet with an address range of
192.168.0.0/24
. -
In the Public IP field, select
Auto
.
-
- Under Access, specify the information required to access the instance:
-
Enter the username in the Login field.
Alert
Do not use the
root
username or other names reserved by the operating system. To perform operations that require superuser permissions, use thesudo
command. -
In the SSH key field, paste the contents of the public key file. You need to create a key pair for the SSH connection yourself. To learn how, see Connecting to a VM via SSH.
-
- Click Create VM.
It may take a few minutes to create the VM. When the VM changes its status to RUNNING
and GitLab starts, you can proceed to setup.
Configure GitLab
To configure GitLab and enable Continuous Integration (CI), create a new project and enter the CI authorization parameters:
-
Log in to the Managed Service for GitLab instance web interface.
-
Click Create a project.
-
Click Create blank project.
-
Fill out the fields below:
- Project name:
gitlab-test
. - Project URL: Select the administrator user in the field next to the Managed Service for GitLab instance FQDN.
Leave the other fields unchanged.
- Project name:
-
Click Create project.
-
On the Yandex Compute Cloud page, select the created VM and copy its public IP.
-
Connect to the VM via SSH.
-
Get the GitLab administrator password using the following VM command:
sudo cat /etc/gitlab/initial_root_password
-
Copy the password (without spaces) from the
Password
row to the clipboard or a separate file. -
Open
http://<VM_public_IP_address>
in your browser. This will take you to the GitLab web interface. -
Log in using the administrator account:
- Username or email:
root
- Password: Password you copied earlier
If you are unable to log in, reset the administrator account password
. - Username or email:
-
Log in to the system again using the administrator account and the new password.
-
Select Create a project.
-
Set the project name:
gitlab-test
. -
Click Create project.
Create a test application
Create a test application that can be deployed in a Managed Service for Kubernetes cluster. To do this, add Dockerfile
to the project:
-
Log in to GitLab.
-
Open the GitLab project.
-
Click
in the repository navigation bar and select New file from the drop-down menu. -
Name the file as
Dockerfile
and add the following code to it:FROM alpine:3.10 CMD echo "Hello"
-
Add a commit comment in the Commit message field:
Dockerfile for a test application
. -
Click Commit changes.
Create a GitLab Runner
To run build tasks in the Yandex Managed Service for Kubernetes cluster, create a GitLab Runner
Once it is installed, you can run automated builds inside your Managed Service for Kubernetes cluster.
For more information about installing and running GitLab Runner, see the GitLab documentation
Create GitLab environment variables
To allow Managed Service for GitLab to save Docker images and their tags in Yandex Container Registry, create the GitLab environment variables
-
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter. -
Create a static key for the
account-for-container-registry
service account you created earlier:yc iam key create --service-account-name account-for-container-registry -o key.json
The key is saved in the
key.json
file in the folder where you ran the command. -
Open your project in GitLab.
-
Go to Settings in the left-hand panel, then select CI/CD from the drop-down list.
-
Click Expand next to Variables.
-
Add environment variables with the protection option disabled:
Variable Its value CI_REGISTRY
cr.yandex/<registry_ID>
. Specify the ID of the Yandex Container Registry registry that you created previously.CI_REGISTRY_KEY
Contents of the key.json
file.To add a variable:
- Click Add variable.
- In the window that opens, enter the variable name in the Key field and the value in the Value field.
- Disable the Protect variable option.
- Click Add variable.
Create the CI script configuration file
To build images from a Dockerfile without Docker, use kaniko
To publish Docker images from your GitLab project in Yandex Container Registry, create a CI script:
-
Open the
gitlab-test
project. -
Click
in the repository navigation bar and select New file from the drop-down menu. -
Name the file as
.gitlab-ci.yml
. Add to it the steps to build a Docker image and push it to Yandex Container Registry:.gitlab-ci.yml
build: stage: build # Using kaniko to create a container inside the container for enhanced security. image: name: gcr.io/kaniko-project/executor:debug entrypoint: [""] script: - mkdir -p /kaniko/.docker # Pushing the container image to the registry. The image is marked with the commit hash. - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json - >- /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}"
This file includes the variables:
CI_REGISTRY
andCI_REGISTRY_KEY
: Added to GitLab at the previous step.CI_PROJECT_DIR
,CI_PROJECT_PATH
, andCI_COMMIT_SHORT_SHA
: Pre-defined in GitLab .
-
Add the following commit comment in the Commit message field:
Create a CI pipeline
. -
Click Commit changes.
Check the result
Each commit is followed by a build script. To check the script execution results:
-
Select Build on the left-hand panel in the
gitlab-test
project, and then select Pipelines from the drop-down menu. -
Make sure that the
build
step has thepassed
status. This means that the CI script has been executed successfully. -
Go to the management console
, then open the Yandex Container Registry registry.If the script completes successfully, a new repository is added to the registry. New Docker images from the GitLab project are added to this repository at each commit.
Enable a Docker image lifecycle policy
To avoid storing outdated Docker images and their tags, configure a Docker image lifecycle policy. The policy manages the images stored in your Container Registry repository to free up its space in a timely manner. This way, you no longer pay for storing outdated images.
To create a policy, follow the guide.
Specifics of working with the policy:
-
External Container Registry and Docker image lifecycle policies affect CI script performance
. -
The policy imposes a limit on the maximum number of images that you can check per policy run. If the number of images in your Container Registry repository exceeds this limit, run the policy several times. You can check all your images in this manner.
Scan your Docker images for vulnerabilities
To detect vulnerabilities in your Docker images, you can additionally activate a vulnerability scanner in Yandex Container Registry. The scanner checks the versions of packages installed in your images against CVE
To enable scanning, expand your GitLab project's CI script:
-
Open the
gitlab-test
project. -
Open the
.gitlab-ci.yml
file. -
Add to it the steps for vulnerability scanning of your Docker image:
.gitlab-ci.yml
stages: - build - test <the_build_section_that_you_previously_added_to_the_file> container_scanning_free_yc: stage: test # Using the jq utility to search for an ID and write logs. image: name: pindar/jq entrypoint: [""] artifacts: when: always paths: - gl-container-scanning-report-yc.json variables: # Specify the ID of the registry you created earlier. CI_REGISTRY_ID: "<registry_ID>" script: - export CI_COMMIT_SHORT_SHA=${CI_COMMIT_SHORT_SHA} # Install Yandex Cloud CLI. - curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash -s -- -a && cp /root/yandex-cloud/bin/yc /usr/bin/ # Start scanning. - echo "Scanning image ${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}..." - export IMAGE_ID=$(yc container image list --registry-id $CI_REGISTRY_ID --format=json | jq -r --arg CI_COMMIT_SHORT_SHA $CI_COMMIT_SHORT_SHA '.[] | select(.tags[0]==$CI_COMMIT_SHORT_SHA) | .id ') # Write logs. - export SCAN_RESULT=$(yc container image scan $IMAGE_ID --format=json) - export CRIT_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.critical // 0') - export HIGH_VULN=$(echo $SCAN_RESULT | jq -r '.vulnerabilities.high // 0') - export SCAN_ID=$(echo $SCAN_RESULT | jq -r '.id') - echo "Scan results:" - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' - yc container image list-vulnerabilities --scan-result-id="${SCAN_ID}" --format json | jq -r '.[] | select(.severity=="CRITICAL", .severity=="HIGH")' > gl-container-scanning-report-yc.json # Check the result. - (( SUM = $CRIT_VULN + $HIGH_VULN )) && (( RES = (SUM >= 1) )) && echo $RES && echo "image has $CRIT_VULN critical vulnerabilities and $HIGH_VULN high vulnerabilities" && exit 1 || echo "image has no high or critical vulnerabilities" exit 0
-
Add the following commit comment in the Commit message field:
Turn on a vulnerability scanner
. -
Click Commit changes. After that, the updated script will run.
To make sure that the image scan was successful:
-
Select Build on the left-hand panel in the
gitlab-test
project, and then select Pipelines from the drop-down menu. -
Make sure that the
build
andtest
steps have thepassed
status. This means that the CI script has been executed successfully. -
Go to the management console
, then open the Yandex Container Registry registry. -
Open your repository with Docker images from the GitLab project.
-
Go to the folder with the GitLab project name.
-
Make sure that the Scan status column includes Ready.
-
In the Date of last scan column, click the link with the scan time.
You will see the scan results. If vulnerabilities were found in the images, you will see them in the results.
Delete the resources you created
If you no longer need the resources you created, delete them:
- Delete the Managed Service for GitLab instance or the created VM with the GitLab image.
- Delete all Docker images from the Container Registry registry.
Delete the other resources depending on how they were created:
- Delete the Managed Service for Kubernetes cluster.
- If you reserved a public IP address for the Managed Service for Kubernetes cluster, delete it.
- Delete the service accounts.
- Delete the Container Registry registry.
- Delete the subnets and network.
To delete the infrastructure created with Terraform:
-
In the terminal window, go to the directory containing the infrastructure plan.
-
Delete the
container-registry-and-gitlab.tf
configuration file. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the resources described in the
container-registry-and-gitlab.tf
configuration file will be deleted. -