Building a CI/CD pipeline using serverless products

You can build a continuous integration / continuous delivery (CI/CD) pipeline using serverless products.

As a project example, we will use a web app based on Django that simulates a shopping cart in an online store. Product descriptions are stored in a database, while the product cart status is stored by the service in a user session. The Django app is deployed in a Serverless Containers container with secrets securely delivered to the app using Yandex Lockbox. Yandex API Gateway accepts user requests and redirects them to the app's container.

The project uses two environments:

• prod: Production environment available to users.
• testing: Test environment used for testing the app before its release into prod.

For each environment, there is an individual folder in Yandex Cloud, as well as a separate set of static resources, such as the DB and service accounts. This isolates the environments from each other at the level of Yandex Identity and Access Management settings.

In addition, there is a common folder named infra. It contains a Yandex Container Registry registry into which all the app's collected Docker images are published. The images are published under a separate service account named builder. The service accounts of the prod and testing environments have limited permissions in the infra folder and are only allowed to pull Docker images.

To build a CI/CD pipeline using serverless products:

1. Create a GitLab instance.
2. Configure GitLab.
3. Create a GitLab Runner.
4. Upload files to the GitLab repository.
5. Create GitLab environment variables.
6. Create the CI script configuration file.
7. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Download a projectDownload a project

Clone the yc-serverless-gitlab-ci-cd repository using Git:

git clone https://github.com/yandex-cloud-examples/yc-serverless-gitlab-ci-cd.git

Install additional dependenciesInstall additional dependencies

Install the following items in the local environment:

• Yandex Cloud command line interface.

• jq JSON stream processor.

• yq YAML stream processor.

• Python 3.8 or higher.

• Python libraries listed in the application/requirements.txt project file:

  python -m pip install -r ./application/requirements.txt
  

Prepare the infrastructurePrepare the infrastructure

1. Go to the project directory and run the bootstrap.sh script by specifying your cloud ID:

   YC_CLOUD_ID=<cloud_ID> ./bootstrap.sh
   

   The script will deploy the basic infrastructure and create YAML files with a description of the resources you created in the config directory. You can edit the script to create additional folders with required resources. For example, you can do it to add another test environment.

   Warning

   Once completed, the script will output environment variables for GitLab and their values. Save them, as you will need them later.

2. Create service accounts:

   • Service account for the resources with the editor role for the folder where the Yandex Managed Service for Kubernetes cluster is created. This service account will be used to create the resources required for the Managed Service for Kubernetes cluster.
   • Service account for nodes with the container-registry.images.puller role for the folder with the Docker image registry. Nodes will pull the required Docker images from the registry on behalf of this account.

   You can use the same service account for both operations.

3. Create a Managed Service for Kubernetes cluster and a node group. When creating the cluster, specify the previously created service accounts for the resources and nodes.

4. Configure security groups for the Managed Service for Kubernetes cluster.

5. Configure a security group for the Yandex Managed Service for GitLab instance.

Create a GitLab instanceCreate a GitLab instance

Create a Managed Service for GitLab instance and a VM with a GitLab image on the same cloud network as the Managed Service for Kubernetes cluster.

Configure GitLabConfigure GitLab

To configure GitLab and enable Continuous Integration (CI), create a new project and enter the CI authorization parameters:

Create a GitLab RunnerCreate a GitLab Runner

To run build tasks in the Yandex Managed Service for Kubernetes cluster, create a GitLab Runner. To do this, install the GitLab Runner application by following this guide.

Once it is installed, you can run automated builds inside your Managed Service for Kubernetes cluster.

For more information about installing and running GitLab Runner, see the GitLab documentation.

Upload files to the GitLab repositoryUpload files to the GitLab repository

1. Add an SSH key to securely access GitLab.

2. Clone the gitlab-test repository using SSH.

3. Copy all files from the yc-serverless-gitlab-ci-cd repository to gitlab-test.

4. Go to the gitlab-test directory.

5. Index the new files:

   git add .
   

6. Commit the files:

   git commit -m "Add project files"
   

7. Push changes to the gitlab-test repository:

   git push
   

Create GitLab environment variablesCreate GitLab environment variables

1. In GitLab, go to Settings in the left-hand panel and select CI/CD from the drop-down list.

2. Click Expand next to Variables.

3. Add environment variables with the protection option disabled. All required variables and their values were output following the execution of the bootstrap.sh script:

   • cloud_id: Your cloud ID.
   • CI_REGISTRY: ID of the Container Registry registry in the infra folder prefixed with the cr.yandex/ prefix.
   • CI_REGISTRY_KEY: builder service account key.
   • cart_prod: Name of the production folder in Yandex Cloud.
   • DOCAPI_ENDPOINT_prod: Document API endpoint of the Yandex Managed Service for YDB database in the prod folder.
   • PROD_SA_ID: The deployer service account ID in the prod folder.
   • SA_PROD_DEPLOYER_PRIVATE_KEY: deployer service account key in the prod folder.
   • prod_container_id: ID of the Serverless Containers container in the prod folder.
   • cart_testing: Name of the test folder in Yandex Cloud.
   • DOCAPI_ENDPOINT_testing: Document API endpoint of the Managed Service for YDB database in the testing folder.
   • TESTING_SA_ID: deployer service account ID in the testing folder.
   • SA_TESTING_DEPLOYER_PRIVATE_KEY: deployer service account key in the testing folder.

   To add a variable:

   1. Click Add variable.
   2. In the window that opens, enter the variable name in the Key field and the value in the Value field.
   3. Disable the Protect variable option.
   4. Click Add variable.

Create the CI script configuration fileCreate the CI script configuration file

1. Open the gitlab-test project.

2. Click in the repository navigation bar and select New file from the drop-down menu.

3. Name the file as .gitlab-ci.yml and add the following build stages to it:

   .gitlab-ci.yml

   stages:
     - build
     - deploy-test-env
     - test
     - delete-test-env
     - release
   
   # Building a container image.
   build-job:
     stage: build
   # Using kaniko to create a container inside the container for enhanced security.
     image:
       name: gcr.io/kaniko-project/executor:debug
       entrypoint: [""]
     script:
       - mkdir -p /kaniko/.docker
   # Upload the container image to the registry. The image is marked with the commit hash.
       - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n "json_key:${CI_REGISTRY_KEY}" | base64 | tr -d '\n' )\"}}}" > /kaniko/.docker/config.json
       - >-
         /kaniko/executor
         --context "${CI_PROJECT_DIR}/application"
         --dockerfile "${CI_PROJECT_DIR}/application/Dockerfile"
         --destination "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}"
   
   # Deploying a test environment using the embedded container image.
   deploy-test-env-job:
     stage: deploy-test-env
     image: alpine:3.15
     script:
   # Installing tools.
       - apk add -q --no-cache bash curl jq gettext
       - apk add yq --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community
       - curl --fail -silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
    # Authentication with a service account key.
       - echo "$SA_TESTING_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
   # Setting up the variables to deploy API Gateway and creating a container.
       - export sa_id=$TESTING_SA_ID
       - export container_id=$(yc serverless container create --name testing --cloud-id ${cloud_id} --folder-name ${cart_testing} | yq .id)
   # Deploying a revision.
       - yc serverless container revision deploy --container-name testing --image "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}" --cores 1 --memory 512mb --concurrency 1 --execution-timeout 10s --cloud-id ${cloud_id} --folder-name ${cart_testing} --service-account-id ${TESTING_SA_ID} --environment DOCAPI_ENDPOINT=${DOCAPI_ENDPOINT_testing},DB_TABLE_PREFIX='' --secret environment-variable=AWS_ACCESS_KEY_ID,name=cart-app,key=AWS_ACCESS_KEY_ID --secret environment-variable=AWS_SECRET_ACCESS_KEY,name=cart-app,key=AWS_SECRET_ACCESS_KEY --secret environment-variable=SECRET_KEY,name=cart-app,key=SECRET_KEY
   # Setting up a template and deploying API Gateway.
       - (cat ${CI_PROJECT_DIR}/apigw.yaml.j2 | envsubst) > apigw.yaml
       - cat apigw.yaml
       - yc serverless api-gateway create --name testing --spec=apigw.yaml --description "created from gitlab CI" --cloud-id ${cloud_id} --folder-name ${cart_testing}
       - mkdir output
       - export gwdomain=$(yc serverless api-gateway get testing --cloud-id ${cloud_id} --folder-name ${cart_testing} | yq .domain)
       - echo "https://"$gwdomain>output/gwurl
     artifacts:
       paths: [output/]
   
   e2e-test-job:
     stage: test
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl
       - apk add yq --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community
       - cat output/gwurl
       - export gwurlvar=$(cat output/gwurl)
       - curl $gwurlvar
   
   load-test-job:
     stage: test
     image: alpine:3.15
     script:
       - echo "Here goes load testing commands"
       - echo "Might even invoke bash with prepared bash script"
       - echo "Hello!"
   
   delete-test-env-job:
     stage: delete-test-env
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl jq gettext yq
       - curl --fail --silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
       - echo "$SA_TESTING_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
       - yc serverless api-gateway delete testing --cloud-id ${cloud_id} --folder-name ${cart_testing}
       - yc serverless container delete testing --cloud-id ${cloud_id} --folder-name ${cart_testing}
   
   release-job:
     stage: release
     image: alpine:3.15
     script:
       - apk add -q --no-cache bash curl jq gettext
       - curl --fail --silent --location -remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
       - bash install.sh -i /usr/local/yandex-cloud -n
       - ln -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
       - echo "$SA_PROD_DEPLOYER_PRIVATE_KEY" > key.json
       - yc config profile create sa-profile
       - yc config set service-account-key key.json
       - yc serverless container revision deploy --container-name prod --image "${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_SHORT_SHA}" --cores 1 --memory 512mb --concurrency 1 --execution-timeout 10s --cloud-id ${cloud_id} --folder-name ${cart_prod} --service-account-id ${PROD_SA_ID} --environment DOCAPI_ENDPOINT=${DOCAPI_ENDPOINT_prod},DB_TABLE_PREFIX='' --secret environment-variable=AWS_ACCESS_KEY_ID,name=cart-app,key=AWS_ACCESS_KEY_ID --secret environment-variable=AWS_SECRET_ACCESS_KEY,name=cart-app,key=AWS_SECRET_ACCESS_KEY --secret environment-variable=SECRET_KEY,name=cart-app,key=SECRET_KEY
       - container_id=${prod_container_id}
   # Creating a production environment.
     environment:
       name: production/$CI_COMMIT_SHORT_SHA
   

4. Add a comment to the commit in the Commit message field: CI scripts.

5. Click Commit changes.

In the .gitlab-ci.yml file, the following steps of the CI script are described:

• build: Building a Docker image using Dockerfile and pushing the image to Container Registry.
• deploy-test-env: Deploying the application in test mode. In addition, the artifacts mechanism for transferring data from one stage to another is described but not used. You can configure it, if required.
• test: Testing the application. The tests involve E2E simulation and load testing. Describe and set up custom tests.
• delete-test-env: Deleting the test app.
• release: Deploying the app in production. This stage also uses deployment environments. They are created and saved each time the pipeline is run successfully. Use them to restore and deploy the previous app version.

After you save the .gitlab-ci.yml configuration file, the build script will start.

Check the resultCheck the result

To check the build script results, select Build on the left-hand panel in the gitlab-test project, and then Pipelines from the drop-down menu. Make sure all the five stages complete successfully.

The app will be available at the URL of the API Gateway API gateway's service domain in the prod folder. You can find the URL in the management console or the domain field in the bootstrap.sh script runtime log.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the folders named prod, testing, and infra together with their resources.
2. Delete the Managed Service for GitLab instance or the created VM with the GitLab image.
3. Delete the Managed Service for Kubernetes cluster.
4. If you reserved a public static IP address for the cluster, delete it.
5. Delete the service accounts.