Building a CI/CD pipeline using serverless products

You can build a continuous integration/continuous delivery (CI/CD) pipeline using serverless products.

As an example, we will use a Django web app simulating an online store shopping cart. The app uses a database to store product descriptions and the user session to store the product cart status. We deploy the Django app in a Serverless Containers container with secrets securely delivered to it through Yandex Lockbox. Yandex API Gateway accepts user requests and redirects them to the app container.

The project uses two environments:

• prod: Production environment available to users.
• testing: Test environment for testing the app before the release.

For each environment, there is an individual Yandex Cloud folder and separate static resources, e.g., the DB and service accounts. This isolates the environments from each other at the Yandex Identity and Access Management settings level.

There is also a common infra folder with a in Yandex Container Registry registry containing all app’s Docker images. The system publishes the images under the builder service account. The prod and testing environment service accounts have restricted permissions in the infra folder limited to pulling Docker images.

To build a CI/CD pipeline with serverless products:

1. Create a GitLab instance.
2. Configure GitLab.
3. Create a GitLab Runner.
4. Upload files to the GitLab repository.
5. Create GitLab environment variables.
6. Create the CI script configuration file.
7. Check the result.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for disks and continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for using the Yandex Managed Service for Kubernetes master (see Managed Service for Kubernetes pricing).
• Fee for storing created Docker images (see Container Registry pricing).
• Fee for storing secrets (see Yandex Lockbox pricing).
• Fee for the number of container calls, computing resources allocated to execute the application, and outgoing traffic (see Serverless Containers pricing).
• Fee for requests to the API gateway (see API Gateway pricing).
• Fee for using public IP addresses (see Yandex Virtual Private Cloud pricing).

Getting startedGetting started

Download a projectDownload a project

Clone the yc-serverless-gitlab-ci-cd repository using Git:

git clone https://github.com/yandex-cloud-examples/yc-serverless-gitlab-ci-cd.git

Install additional dependenciesInstall additional dependencies

Install the following tools in the local environment:

• Yandex Cloud command line interface.

• jq JSON stream processor.

• yq YAML stream processor.

• Python 3.8 or higher.

• Python libraries listed in the application/requirements.txt project file:

  python -m pip install -r ./application/requirements.txt
  

Prepare the infrastructurePrepare the infrastructure

1. Navigate to the project directory; run the bootstrap.sh script and specify your cloud ID:

   YC_CLOUD_ID=<cloud_ID> ./bootstrap.sh
   

   The script will deploy the basic infrastructure and create YAML resource description files in the config directory. You can edit the script to create additional resource folders. For example, this way you can add another test environment.

   Warning

   Once completed, the script will display GitLab environment variables and their values. Save them, as you will need them later.

2. Create service accounts:

   • Service account for resources with the editor role for the Yandex Managed Service for Kubernetes cluster folder. This service account will be used to create Managed Service for Kubernetes cluster resources.
   • Service account for nodes with the container-registry.images.puller role for the Docker image registry folder. Nodes will pull the required Docker images from the registry on behalf of this account.

   You can use the same service account for both operations.

3. Create a Managed Service for Kubernetes cluster and a node group. When creating the cluster, specify the previously created service accounts for the resources and nodes.

4. Configure security groups for the Managed Service for Kubernetes cluster.

5. Configure a security group for the Yandex Managed Service for GitLab instance.

Create a GitLab instanceCreate a GitLab instance

Create either a Managed Service for GitLab instance or a VM with a GitLab image on the same cloud network as the Managed Service for Kubernetes cluster.

Configure GitLabConfigure GitLab

To configure GitLab and enable Continuous Integration (CI), create a new project and enter the CI authorization parameters:

Create a GitLab RunnerCreate a GitLab Runner

To run build tasks in the Yandex Managed Service for Kubernetes cluster, create a GitLab Runner. To do this, install GitLab Runner by following this guide.

Once it is installed, you can run automated builds inside your Managed Service for Kubernetes cluster.

For more information about installing and running GitLab Runner, see the GitLab documentation.

Upload files to the GitLab repositoryUpload files to the GitLab repository

1. Add an SSH key to securely access GitLab.

2. Clone the gitlab-test repository over SSH.

3. Copy all files from the yc-serverless-gitlab-ci-cd repository to gitlab-test.

4. Navigate to the gitlab-test directory.

5. Index the new files:

   git add .
   

6. Commit the files:

   git commit -m "Add project files"
   

7. Push changes to the gitlab-test repository:

   git push
   

Create GitLab environment variablesCreate GitLab environment variables

1. In GitLab, navigate to Settings in the left-hand panel and select CI/CD from the drop-down list.

2. Click Expand next to Variables.

3. Add environment variables with the protection option disabled. The bootstrap.sh script will display all the required variables and their values:

   • cloud_id: Your cloud ID.
   • CI_REGISTRY: Container Registry-based registry ID with the cr.yandex/ prefix in the infra folder.
   • CI_REGISTRY_KEY: The builder service account key.
   • cart_prod: Yandex Cloud production folder name.
   • DOCAPI_ENDPOINT_prod: Document API endpoint of the Yandex Managed Service for YDB database in the prod folder.
   • PROD_SA_ID: deployer service account ID in the prod folder.
   • SA_PROD_DEPLOYER_PRIVATE_KEY: deployer service account key in the prod folder.
   • prod_container_id: Serverless Containers-based container ID in the prod folder.
   • cart_testing: Yandex Cloud test folder name.
   • DOCAPI_ENDPOINT_testing: Document API endpoint of the Managed Service for YDB database in the testing folder.
   • TESTING_SA_ID: The deployer service account ID in the testing folder.
   • SA_TESTING_DEPLOYER_PRIVATE_KEY: The deployer service account key in the testing folder.

   To add a variable:

   1. Click Add variable.
   2. In the window that opens, specify the variable name in the Key field and its value in the Value field.
   3. Disable the Protect variable option.
   4. Click Add variable.

Create the CI script configuration fileCreate the CI script configuration file

1. Open the gitlab-test project.

2. Click in the repository navigation bar and select New file from the drop-down menu.

3. Name the file as .gitlab-ci.yml and add the following build stages to it:

   .gitlab-ci.yml

   stages:
     - build
     - deploy-test-env
     - test
     - delete-test-env
     - release
   
   # Building a container image.
   build-job:
     stage: build
   # Using `kaniko` to create a container inside another container for enhanced security.
     image:
       name: gcr.io/kaniko-project/executor:debug
       entrypoint: [""]
     script:
       - mkdir -p /kaniko/.docker
   # Upload the container image to the registry. The image is tagged with the commit hash.
       - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
       - >-
         /kaniko/executor
         --context "${CI_PROJECT_DIR}/application"
         --dockerfile "${CI_PROJECT_DIR}/application/Dockerfile"
         --destination "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}"
   
   # Deploying a test environment using a built-in container image.
   deploy-test-env-job:
     stage: deploy-test-env
     image: alpine:3.15
     script:
   # Installing tools.
       - apk add -q --no-cache bash curl jq gettext
       - apk add yq --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community
       - curl --fail -silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
    # Authentication with a service account key.
       - echo "$SA_TESTING_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
   # Setting up the API Gateway variables and creating a container.
       - export sa_id=$TESTING_SA_ID
       - export container_id=$(yc serverless container create --name testing --cloud-id ${cloud_id} --folder-name ${cart_testing} | yq .id)
   # Deploying a revision.
       - yc serverless container revision deploy --container-name testing --image "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}" --cores 1 --memory 512mb --concurrency 1 --execution-timeout 10s --cloud-id ${cloud_id} --folder-name ${cart_testing} --service-account-id ${TESTING_SA_ID} --environment DOCAPI_ENDPOINT=${DOCAPI_ENDPOINT_testing},DB_TABLE_PREFIX='' --secret environment-variable=AWS_ACCESS_KEY_ID,name=cart-app,key=AWS_ACCESS_KEY_ID --secret environment-variable=AWS_SECRET_ACCESS_KEY,name=cart-app,key=AWS_SECRET_ACCESS_KEY --secret environment-variable=SECRET_KEY,name=cart-app,key=SECRET_KEY
   # Setting up a template and deploying API Gateway.
       - (cat ${CI_PROJECT_DIR}/apigw.yaml.j2 | envsubst) > apigw.yaml
       - cat apigw.yaml
       - yc serverless api-gateway create --name testing --spec=apigw.yaml --description "created from gitlab CI" --cloud-id ${cloud_id} --folder-name ${cart_testing}
       - mkdir output
       - export gwdomain=$(yc serverless api-gateway get testing --cloud-id ${cloud_id} --folder-name ${cart_testing} | yq .domain)
       - echo "https://"$gwdomain>output/gwurl
     artifacts:
       paths: [output/]
   
   e2e-test-job:
     stage: test
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl
       - apk add yq --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community
       - cat output/gwurl
       - export gwurlvar=$(cat output/gwurl)
       - curl $gwurlvar
   
   load-test-job:
     stage: test
     image: alpine:3.15
     script:
       - echo "Here goes load testing commands"
       - echo "Might even invoke bash with prepared bash script"
       - echo "Hello!"
   
   delete-test-env-job:
     stage: delete-test-env
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl jq gettext yq
       - curl --fail --silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
       - echo "$SA_TESTING_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
       - yc serverless api-gateway delete testing --cloud-id ${cloud_id} --folder-name ${cart_testing}
       - yc serverless container delete testing --cloud-id ${cloud_id} --folder-name ${cart_testing}
   
   release-job:
     stage: release
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl jq gettext
       - curl --fail --silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
       - echo "$SA_PROD_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
       - yc serverless container revision deploy --container-name prod --image "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}" --cores 1 --memory 512mb --concurrency 1 --execution-timeout 10s --cloud-id ${cloud_id} --folder-name ${cart_prod} --service-account-id ${PROD_SA_ID} --environment DOCAPI_ENDPOINT=${DOCAPI_ENDPOINT_prod},DB_TABLE_PREFIX='' --secret environment-variable=AWS_ACCESS_KEY_ID,name=cart-app,key=AWS_ACCESS_KEY_ID --secret environment-variable=AWS_SECRET_ACCESS_KEY,name=cart-app,key=AWS_SECRET_ACCESS_KEY --secret environment-variable=SECRET_KEY,name=cart-app,key=SECRET_KEY
       - container_id=${prod_container_id}
   # Creating a production environment.
     environment:
       name: production/$CI_COMMIT_SHORT_SHA
   

4. Add a comment in the Commit message field: CI scripts.

5. Click Commit changes.

The .gitlab-ci.yml file contains the following steps of the CI script:

• build: Building a Docker image with Dockerfile and pushing the image to Container Registry.
• deploy-test-env: App test deployment. Additionally, we described the artifacts feature to transfer data from one stage to another that we are not going to use here. You can configure it, if required.
• test: Testing the app. The tests involve E2E simulation and load testing. You can describe and set up custom tests as well.
• delete-test-env: Deleting the test app.
• release: Deploying the app in production. This stage also uses deployment environments. The system creates and saves them each time the pipeline is run successfully. Use them to restore and deploy the previous app version.

After you save the .gitlab-ci.yml configuration file, the build script will start.

Check the resultCheck the result

To check the build script results, select Build on the left-hand panel in the gitlab-test project, and then Pipelines from the drop-down menu. Make sure all five stages complete successfully.

You can access the app at the API Gateway service domain URL in the prod folder. You can find this URL in the management console or in the domain field of the bootstrap.sh script runtime log.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the prod, testing, and infra folders with their content.
2. Delete the Managed Service for GitLab instance or the GitLab image VM.
3. Delete the Managed Service for Kubernetes cluster.
4. Delete the cluster public static IP address if you reserved one.
5. Delete the service accounts.