Invoking load testing from GitLab CI

GitLab is a web-based Git repository management site and system. GitLab also lets developers run a continuous process for writing, testing, and deploying code.

In this tutorial, you will add the stage of invoking a load test of the application being deployed using Load Testing to the continuous integration and continuous delivery (CI/CD) pipeline. During this stage, the script will create a test agent, run the test, and check the test result.

To add the load testing invocation from GitLab CI:

1. Prepare your cloud.
2. Prepare your infrastructure.
3. Prepare a file with test data.
4. Create GitLab environment variables.
5. Add the load testing stage to the CI script configuration file.

If you no longer need the resources you created, delete them.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

1. If the agent is hosted on Yandex Cloud, you pay for computing resources (see Yandex Compute Cloud pricing).

2. Fee for data storage in a bucket and operations with data (see Object Storage pricing).

Prepare the infrastructurePrepare the infrastructure

Create a service accountCreate a service account

1. Create a service account named sa-loadtest in the folder to host the agent that will generate the load.

2. Assign the following roles to the service account:

   • loadtesting.generatorClient
   • loadtesting.loadTester
   • iam.serviceAccounts.user
   • compute.editor
   • vpc.user
   • vpc.publicAdmin (optional, if deploying the agent in a public network).

3. Create an authorized key for the service account and save it to the key.json file:

Configure a networkConfigure a network

Create and configure a NAT gateway in the subnet where your test target is and where the agent will reside. This will enable the agent to access Load Testing.

Configure the security groupConfigure the security group

Set up the test agent's security group:

1. Create an agent security group named agent-sg.
2. Add rules:

   1. Rule for outgoing HTTPS traffic to the Load Testing public API:

      • Port range: 443
      • Protocol: TCP
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

      This will allow connecting the agent to Load Testing to manage tests from the interface and get test results.

   2. Rule for incoming SSH traffic:

      • Port range: 22
      • Protocol: TCP
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

      This will allow you to connect to the agent over SSH and manage tests from the console or collect debugging information.

   3. Rule for outgoing traffic when generating load to the test target:

      • Port range: 0-65535
      • Protocol: Any
      • Destination name: Security group
        Select From list. Specify the security group where the test target is located.

      Create this rule for each test target with a unique security group.

Prepare a CI/CD pipeline in GitLab for deploying the test targetPrepare a CI/CD pipeline in GitLab for deploying the test target

In this tutorial, we will use an application with the 51.250.103.44 public IP address as a test target example. For an example of building a CI/CD pipeline in GitLab, see this article.

Prepare a file with test dataPrepare a file with test data

1. Generate payloads in HTTP_JSON format:

   {"host": "51.250.103.44", "method": "GET", "uri": "/", "tag": "url1", "headers": {"User-agent": "Tank", "Connection": "Close"}}
   

   Where:

   • host: Host header value
   • method: HTTP request method
   • uri: Request URI
   • tag: Request tag to display in reports
   • headers: Request headers

You can also use the Dispatcher tool to prepare payloads.

1. Save the payloads to a file named httpjson.payload.

2. Create a bucket and upload the test data file into it.

3. Grant read permissions for the bucket to the service account. To do this, edit the bucket's ACL:

   1. In the management console, select the folder where the bucket is located.
   2. In the list of services, select Object Storage.
   3. Click next to the bucket and select Bucket ACL.
   4. In the window that opens, enter the service account name, select the READ permissions for the bucket, and click Add.
   5. Click Save.

   This grants the service account permission to read data from this bucket only.

Create GitLab environment variablesCreate GitLab environment variables

1. In GitLab, go to Settings in the left-hand panel and select CI/CD from the drop-down list.

2. Click Expand next to Variables.

3. Add environment variables with the protection option disabled:

   • SERVICE_ACCOUNT_ID: ID of the sa-loadtest service account
   • SA_AUTHORIZED_KEY: Contents of the authorized key for the sa-loadtest service account
   • SECURITY_GROUP_ID: ID of the agent-sg security group of the test agent
   • SUBNET_ID: ID of the subnet to host the test agent
   • ZONE: Availability zone for the test agent, e.g.,ru-central1-a
   • YC_FOLDER_ID: ID of the folder for the resources

   To add a variable:

   1. Click Add variable.
   2. In the window that opens, enter the variable name in the Key field and the value in the Value field.
   3. Disable the Protect variable option.
   4. Click Add variable.

Add the load testing stage to the CI script configuration fileAdd the load testing stage to the CI script configuration file

1. Add the test-config.yaml load testing configuration file to your project repository in GitLab:

   pandora:
      enabled: true
      package: yandextank.plugins.Pandora
      config_content:
         pools:
            - id: HTTP
            gun:
               type: http
               target: 51.250.103.44
               ssl: false
            ammo:
               type: http/json
               file: httpjson.payload
            result:
               type: phout
               destination: ./phout.log
            startup:
               type: once
               times: 1000
            rps:
               - type: line
                  from: 0
                  to: 500
                  duration: 60s
            discard_overflow: true
         log:
            level: error
         monitoring:
            expvar:
               enabled: true
               port: 1234
   autostop:
      enabled: true
      package: yandextank.plugins.Autostop
      autostop:
         - limit(5m) # Required parameter
         - quantile(50,100,5,)
   core: {}
   

   Specify the address of your application in the target field. This testing configuration will generate a linear load from 0 to 500 requests per second for 60 seconds. The autostop criterion is also configured. This criterion stops the test if the 50th percentile exceeds 100 milliseconds for 5 seconds.

2. Add to the .gitlab-ci.yml file the load testing stage following the deployment of the application:

   stages:
      ...
      - loadtesting
      ...
   
   loadtesting-job:
      stage: loadtesting
      allow_failure: true
      artifacts:
         when: always
         name: "tests_output"
         paths:
            - agent_id.txt
      before_script:
         # Installing tools.
         - sudo apt-get install -y jq
         - curl -f -s -LO https://storage.yandexcloud.net/yandexcloud-yc/install.sh
         - sudo bash install.sh -i /usr/local/yandex-cloud -n
         - sudo ln -f -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
         # Authenticating with the service account key
         - echo "$SA_AUTHORIZED_KEY" > key.json
         - yc config profile create sa-profile || echo "Profile already exists"
         - yc config set service-account-key key.json
         - yc config set folder-id ${YC_FOLDER_ID}
      script:
         # Creating a testing agent
         - agent_id=$(yc loadtesting agent create --name agent${CI_JOB_ID} --zone ${ZONE} --service-account-id ${SERVICE_ACCOUNT_ID} --network-interface security-group-ids=${SECURITY_GROUP_ID},subnet-id=${SUBNET_ID} --format json | jq -r .id)
         - echo $agent_id > agent_id.txt
         # Awaiting the agent's readiness
         - |
           while true; do
              if ! _status=$(yc loadtesting agent get "$agent_id" --format json | jq -r '.status'); then
                 sleep 10
                 continue
              fi
              if [[ "$_status" == "READY_FOR_TEST" ]]; then
                 echo "Agent is ready"
                 break
              fi
              echo "Waiting for agent..."
              sleep 10
           done
         # Creating a configuration
         - config_id=$(yc loadtesting test-config create --from-yaml-file test-config.yaml --format json | jq -r .id)
         # Running the test
         - test_id=$(yc loadtesting test create --name "test_${CI_COMMIT_SHORT_SHA}" --description "${CI_COMMIT_MESSAGE}" --labels auto=true --configuration agent-id=$agent_id,id=$config_id,test-data=httpjson.payload --test-data name=httpjson.payload,s3bucket=payload-bucket,s3file=httpjson.payload --wait --format json | jq -r .id)
         - yc loadtesting test wait --idle-timeout 120s $test_id
         # Evaluating the test results
         - q50=$(yc loadtesting test get-report-table $test_id --format json | jq -r .overall.quantiles.q50)
         - |
           if [[ $q50 -gt 100 ]]; then
              echo "Q50 exceeded 100 ms, the test was not passed"
              exit 1
           fi
      after_script:
         # Deleting the test agent
         - agent_id=$(cat agent_id.txt)
         - yc loadtesting agent delete $agent_id
         - rm agent_id.txt
   

   During the stage described here, the script will create a test agent, run the test, and check the test result. The result evaluation is based on the 50th percentile. If it exceeds 100ms, the stage will terminate with an error.

   After you save the .gitlab-ci.yml configuration file, the build script will start.

   You can view the test results in more detail in the management console:

   1. In the management console, select Load Testing.
   2. In the left-hand panel, select Tests.
   3. Select the test you created and navigate to the Test results tab.

   You can use any connection logic for this stage:

   • On commits to the main branch
   • On pull request updates

   For more complex scenarios, see the video:
   

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the service accounts.
2. Delete the Object Storage bucket.
3. Make sure that the test agent created by the script was deleted. You can delete the agent manually.
4. Delete the route table.
5. Delete the NAT gateway.