Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Invoking load testing from GitLab CI

Written by
Updated at November 8, 2024

GitLab is a web-based Git repository management site and system. GitLab also lets developers run a continuous process for writing, testing, and deploying code.

In this scenario, you will use Load Testing to add the stage of invoking a load test of the application being deployed to the continuous integration and continuous delivery (CI/CD) pipeline. During this stage, the script will create a test agent, run the test, and check the test result.

To add the load testing invocation from GitLab CI:

  1. Prepare your cloud.
  2. Prepare your infrastructure.
  3. Prepare a file with test data.
  4. Create GitLab environment variables.
  5. Add the load testing stage to the CI scenario configuration file.

If you no longer need the resources you created, delete them.

Prepare your cloud

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

  1. If the agent is hosted on Yandex Cloud, you pay for computing resources (see Yandex Compute Cloud pricing).

  2. Fee for data storage in a bucket and operations with data (see Object Storage pricing).

Prepare the infrastructure

Create a service account

  1. Create a service account named sa-loadtest in the folder that will host the agent to supply the load.

  2. Assign the following roles to the service account:

    • loadtesting.generatorClient
    • loadtesting.loadTester
    • iam.serviceAccounts.user
    • compute.editor
    • vpc.user
    • vpc.publicAdmin (optional, if deploying the agent in a public network).

  3. Create an authorized key for the service account and save it to the key.json file:

Configure a network

Create and configure a NAT gateway in the subnet where your test target is and where the agent will reside. This will enable the agent to access Load Testing.

Configure the security group

Set up the test agent's security group:

  1. Create an agent security group named agent-sg.
  2. Add rules:

    1. Rule for outgoing HTTPS traffic to the Load Testing public API:

      • Port range: 443
      • Protocol: TCP
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

      This will allow connecting the agent to Load Testing to manage tests from the interface and get test results.

    2. Rule for incoming SSH traffic:

      • Port range: 22
      • Protocol: TCP
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

      This will allow you to connect to the agent over SSH and manage tests from the console or collect debugging information.

    3. Rule for outgoing traffic when generating load to the test target:

      • Port range: 0-65535
      • Protocol: Any
      • Destination name: Security group
        Select From list. Specify the security group where the test target is located.

      Create this rule for each test target with a unique security group.

Prepare a CI/CD pipeline in GitLab for deploying the test target

In this scenario, we will use an application with the 51.250.103.44 public IP address to exemplify the test objective. For an example of building a CI/CD pipeline in GitLab, see this article.

Prepare a file with test data

  1. Generate payloads in HTTP_JSON format:

    {"host": "51.250.103.44", "method": "GET", "uri": "/", "tag": "url1", "headers": {"User-agent": "Tank", "Connection": "Close"}}

    Where:

    • host: Host header value.
    • method: HTTP request method.
    • uri: Request URI.
    • tag: Request tag for use in reports.
    • headers: Request headers.

You can also use the Dispatcher tool to prepare payloads.

  1. Save the payloads to a file named httpjson.payload.

  2. Create a bucket and upload the test data file into it.

  3. Grant read permissions for the bucket to the service account. To do this, edit the bucket's ACL:

    1. In the management console, select the folder the bucket is in.
    2. In the list of services, select Object Storage.
    3. Click next to the bucket and select Bucket ACL.
    4. In the window that opens, enter the service account name, select READ for bucket permissions, and click Add.
    5. Click Save.

    This grants the service account permission to read data from this bucket only.

Create GitLab environment variables

  1. In GitLab, go to Settings in the left-hand panel and select CI/CD from the drop-down list.

  2. Click Expand next to Variables.

  3. Add environment variables with the protection option disabled:

    • SERVICE_ACCOUNT_ID: sa-loadtest service account ID.
    • SA_AUTHORIZED_KEY: Contents of the authorized key for the sa-loadtest service account.
    • SECURITY_GROUP_ID: Test agent's agent-sg security group ID.
    • SUBNET_ID: ID of the subnet to host the test agent.
    • ZONE: Availability zone to host the test agent, e.g.,ru-central1-a.
    • YC_FOLDER_ID: ID of the folder to host the resources.

    To add a variable:

    1. Click Add variable.
    2. In the window that opens, enter the variable name in the Key field and the value in the Value field.
    3. Disable the Protect variable option.
    4. Click Add variable.

Add the load testing stage to the CI scenario configuration file

  1. Add the test-config.yaml load testing configuration file to your project repository in GitLab:

    pandora:
   enabled: true
   package: yandextank.plugins.Pandora
   config_content:
      pools:
         - id: HTTP
         gun:
            type: http
            target: 51.250.103.44
            ssl: false
         ammo:
            type: http/json
            file: httpjson.payload
         result:
            type: phout
            destination: ./phout.log
         startup:
            type: once
            times: 1000
         rps:
            - type: line
               from: 0
               to: 500
               duration: 60s
         discard_overflow: true
      log:
         level: error
      monitoring:
         expvar:
            enabled: true
            port: 1234
autostop:
   enabled: true
   package: yandextank.plugins.Autostop
   autostop:
      - limit(5m) # Required parameter
      - quantile(50,100,5,)
core: {}

    In the target field, enter the address of your application. This testing configuration will generate a linear load from 0 to 500 requests per second for 60 seconds. The autostop criterion is also configured. This criterion stops the test if the 50th percentile exceeds 100 milliseconds for 5 seconds.

  2. Add a load testing stage to the .gitlab-ci.yml file after deploying the application:

    stages:
   ...
   - loadtesting
   ...

loadtesting-job:
   stage: loadtesting
   allow_failure: true
   artifacts: 
      when: always
      name: "tests_output"
      paths:
         - agent_id.txt
   before_script:
      # Installing tools.
      - sudo apt-get install -y jq
      - curl --fail --silent --location --remote-name https://storage.yandexcloud.net/yandexcloud-yc/install.sh
      - sudo bash install.sh -i /usr/local/yandex-cloud -n
      - sudo ln -f -s /usr/local/yandex-cloud/bin/yc /usr/local/bin/yc
      # Authentication using a service account key
      - echo "$SA_AUTHORIZED_KEY" > key.json
      - yc config profile create sa-profile || echo "Profile already exists"
      - yc config set service-account-key key.json
      - yc config set folder-id ${YC_FOLDER_ID}
   script:
      # Creating a test agent
      - agent_id=$(yc loadtesting agent create --name agent${CI_JOB_ID} --zone ${ZONE} --service-account-id ${SERVICE_ACCOUNT_ID} --network-interface security-group-ids=${SECURITY_GROUP_ID},subnet-id=${SUBNET_ID} --format json | jq -r .id)
      - echo $agent_id > agent_id.txt
      # Waiting for the agent to become available
      - |
        while true; do
           if ! _status=$(yc loadtesting agent get "$agent_id" --format json | jq -r '.status'); then
              sleep 10
              continue
           fi
           if [[ "$_status" == "READY_FOR_TEST" ]]; then
              echo "Agent is ready"
              break
           fi
           echo "Waiting for agent..."
           sleep 10
        done
      # Creating a configuration
      - config_id=$(yc loadtesting test-config create --from-yaml-file test-config.yaml --format json | jq -r .id)
      # Running the test
      - test_id=$(yc loadtesting test create --name "test_${CI_COMMIT_SHORT_SHA}" --description "${CI_COMMIT_MESSAGE}" --labels auto=true --configuration agent-id=$agent_id,id=$config_id,test-data=httpjson.payload --test-data name=httpjson.payload,s3bucket=payload-bucket,s3file=httpjson.payload --wait --format json | jq -r .id)
      - yc loadtesting test wait --idle-timeout 120s $test_id
      # Evaluating test results
      - q50=$(yc loadtesting test get-report-table $test_id --format json | jq -r .overall.quantiles.q50)
      - |
        if [[ $q50 -gt 100 ]]; then
           echo "Q50 exceeds 100 ms, test failed"
           exit 1
        fi
   after_script:
      # Deleting a test agent
      - agent_id=$(cat agent_id.txt)
      - yc loadtesting agent delete $agent_id
      - rm agent_id.txt

    During the stage described here, the script will create a test agent, run the test, and check the test result. The result evaluation is based on the 50th percentile. If it exceeds 100ms, the stage will terminate with an error.

    After you save the .gitlab-ci.yml configuration file, the build scenario will start.

    You can view the test results in more detail in the management console:

    1. In the management console, select Load Testing.
    2. In the left-hand panel, select Tests.
    3. Select the test you created and navigate to the Test results tab.

    You can use any connection logic for this stage:

    • On commits to the main branch
    • On pull request updates

    For more complex scenarios, see this video:

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. Delete the service accounts.
  2. Delete the Object Storage bucket.
  3. Make sure that the test agent created by the script was deleted. You can delete the agent manually.
  4. Delete the route table.
  5. Delete the NAT gateway.
Previous
Integration with Yandex Tracker
Next
Resource relationships