Comparing load test results

Written by

Yandex Cloud

Updated at June 9, 2025

Get your cloud ready
- Required paid resources
Set up your infrastructure
Prepare a test target
Create a test agent
Prepare a file with test data
Run the test
Repeat the test
Compare the results
How to delete the resources you created

In this tutorial, you will set up a load testing environment, run the tests, and compare their results.

The tutorial uses the Pandora load generator and a simple web service as the test target.

To perform load testing and compare test results:

If you no longer need the resources you created, delete them.

Get your cloud ready

Navigate to the management console and log in to Yandex Cloud or register a new account.
On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resources

If the agent is hosted on Yandex Cloud, you pay for computing resources (see Yandex Compute Cloud pricing).

At the Preview stage, Load Testing is free of charge.

Set up your infrastructure

Create a service account

Create a service account named sa-loadtest in the folder that will host the agent to generate the load.
Assign the loadtesting.generatorClient role to the service account.

Configure your network

Create and configure a NAT gateway in the subnet where your test target and the agent will be placed. This will enable the agent to access Load Testing.

Configure security groups

Configure the test agent security group:
1. Create an agent security group named agent-sg.
2. Add rules:
  1. Rule for outbound HTTPS traffic to the Load Testing public API:
    - Port range: 443.
    - Protocol: TCP.
    - Destination name: CIDR.
    - CIDR blocks: 0.0.0.0/0.
    This will allow you to connect the agent to Load Testing to manage tests from the interface and get test results.
  2. Inbound SSH traffic rule:
    - Port range: 22.
    - Protocol: TCP.
    - Destination name: CIDR.
    - CIDR blocks: 0.0.0.0/0.
    This will allow you to connect to the agent over SSH and manage tests from the console or collect debugging information.
  3. Rule for outbound traffic when generating load to the test target:
    - Port range: 0-65535.
    - Protocol: Any.
    - Destination name: Security group.
      Select From list. Specify the security group comprising the test target.
    Create this rule for each test target with a unique security group.
Configure the test target security group:
1. Create a test target security group named load-target-sg.
2. Add a rule for inbound traffic when generating load to the test target:
  - Port range: 0-65535.
  - Protocol: Any.
  - Destination name: Security group.
    Select From list. Specify the security group comprising the test target.
  This rule allows agents to generate load to the test target or use additional monitoring tools.

Prepare a test target

To prepare a test target:

Create a VM from a public Linux image:
1. Enter load-target as the name.
2. Specify a username, e.g., ycuser.
3. Specify the load-target-sg security group in the network settings.
Connect to the VM over SSH by running this command:
```
ssh ycuser@<VM_public_IP_address>
```
Create a web service file by running:
```
mkdir app
cd app
nano app.py
```

Paste the following code to the web service file:

from flask import Flask, make_response
from flask_cors import CORS
import os

app = Flask(__name__)
CORS(app)

@app.route('/')
def index():
	 res = make_response({"message":"Root URL"}, status_code=200)
	 return res

@app.route('/test')
def get_test():
	 res = make_response({"message":"Get test URL"}, status_code=200)
	 return res

# Run application
if __name__ == '__main__':
	 app.run(host='0.0.0.0', debug=False, port=443)

Install the required packages:
```
pip install flask flask-cors
```
Run the service:
```
python3 app.py
```
The service will expect requests on port 443. Make sure the port is open and set up for HTTPS requests.

Create a test agent

If you do not have an SSH key pair yet, create one.
Create an agent:
Management console

CLI
1. In the management console, select the folder where you want to create the agent.
2. In the list of services, select Load Testing.
3. In the Agents tab, click Create agent.
4. Enter a name for the agent, e.g., agent-008.
5. Specify the same availability zone as the one hosting your test target.
6. Under Agent:
  - Select the appropriate agent type. For more information, see Agent performance.
  - Specify the subnet hosting your test target. Make sure you created and set up a NAT gateway in that subnet.
  - If you have access to security groups, select a preset agent security group.
7. Under Access, specify the agent access credentials:
  - Select the sa-loadtest service account.
  - Under Login, enter the username.
    
    Alert
    
    Do not use root or other reserved usernames. To perform operations requiring root privileges, use the sudo command.
  - In the SSH key field, paste the contents of the public key file.
8. Click Create.
9. Wait for the VM to create. Make sure the agent status has changed to Ready for test.
  Note
  
  If the agent creation process has stopped at Initializing connection, make sure the following conditions are met:
  
  The agent has a public IP address and access to loadtesting.api.cloud.yandex.net:443.
  
  The target subnet has a configured NAT gateway.
  
  The service account assigned to the agent has the required roles.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
1. See the description of the CLI command for creating an agent:
  yc loadtesting agent create --help
2. Select the same availability zone as the one hosting your test target.
3. Select the subnet hosting your test target. Make sure you created and set up a NAT gateway in that subnet.
  
  To get a list of available subnets using the CLI, run this command:
  yc vpc subnet list
  Result:
  +----------------------+---------------------------+----------------------+----------------+-------------------+-----------------+ | ID | NAME | NETWORK ID | ROUTE TABLE ID | ZONE | RANGE | +----------------------+---------------------------+----------------------+----------------+-------------------+-----------------+ | e2lfkhps7bol******** | default-ru-central1-b | enpnf7hajqmd******** | | ru-central1-b | [10.129.0.0/24] | | e9bgnq1bggfa******** | default-ru-central1-a | enpnf7hajqmd******** | | ru-central1-a | [10.128.0.0/24] | | fl841n5ilklr******** | default-ru-central1-d | enpnf7hajqmd******** | | ru-central1-d | [10.130.0.0/24] | +----------------------+---------------------------+----------------------+----------------+-------------------+-----------------+
4. Select the security group. Make sure to configure the security group in advance.
  
  To get a list of available security groups using the CLI, run this command:
  yc vpc security-group list
  Result:
  +----------------------+---------------------------------+--------------------------------+----------------------+ | ID | NAME | DESCRIPTION | NETWORK-ID | +----------------------+---------------------------------+--------------------------------+----------------------+ | enp414a2tnnp******** | default-sg-enpnf7hajqmd******** | Default security group for | enpnf7hajqmd******** | | | | network | | | enpctpve7951******** | sg-load-testing-agents | | enpnf7hajqmd******** | | enpufo9ms0gi******** | sg-load-testing-targets | | enpnf7hajqmd******** | +----------------------+---------------------------------+--------------------------------+----------------------+
5. Get the ID of sa-loadtest service account, specifying its name:
  yc iam service-account get sa-loadtest
  Result:
  id: ajespasg04oc******** folder_id: b1g85uk96h3f******** created_at: "2024-12-04T17:38:57Z" name: sa-loadtest last_authenticated_at: "2024-12-12T19:10:00Z"
6. Create an agent in the default folder:
  yc loadtesting agent create \ --name agent-008 \ --labels origin=default,label-key=label-value \ --zone default-ru-central1-a \ --network-interface subnet-id=e9bgnq1bggfa********,security-group-ids=enpctpve7951******** \ --cores 2 \ --memory 2G \ --service-account-id ajespasg04oc******** --metadata-from-file user-data=metadata.yaml
  Where:
  - --name: Agent name.
  - --labels: Agent labels.
  - --zone: Availability zone to host the agent.
  - --network-interface: Agent network interface settings:
    
    subnet-name: ID of the selected subnet.
    
    security-group-ids: Security group IDs.
  - --cores: Number of CPU cores the agent can use.
  - --memory: Amount of RAM allocated to the agent.
  - --service-account-id: Service account ID.
  - --metadata-from-file: <key>=<value> pair with the name of the file containing the public SSH key path. For an example of the metadata.yaml configuration file, see VM metadata.
  For more information on how to create an agent with the CLI, see the Yandex Cloud Examples repository.
Assign a public IP address to your agent to enable access over SSH:
Management console

CLI
1. In the management console, select the folder with the agent.
2. Select Compute Cloud.
3. Select the agent-008 VM.
4. Under Network interface, in the top-right corner, click and select Add public IP address.
5. In the window that opens:
  - In the Public address field, select Auto.
  - Click Add.
To assign a public IP address to an agent, run the following CLI command:
```
yc compute instance add-one-to-one-nat \
  --id=<VM_ID> \
  --network-interface-index=<VM_network_interface_number> \
  --nat-address=<IP_address>
```
Where:
- --id: VM ID. You can get a list of available VM IDs in the folder using the yc compute instance list CLI command.
- --network-interface-index: VM network interface number. The default value is 0. To get a list of VM network interfaces and their numbers, run yc compute instance get <VM_ID>.
- --nat-address: Public IP address to assign to the VM. This is an optional setting. If you skip it, the VM will get a public IP address automatically.
  
  You can get a list of reserved public IP addresses available in the folder using the yc vpc address list CLI command. The IP address and the VM must be in the same availability zone.
Here is a possible use case:
```
yc compute instance add-one-to-one-nat \
  --id=fhmsbag62taf******** \
  --network-interface-index=0 \
  --nat-address=51.250.*.***
```
Result:
```
id: fhmsbag62taf********
folder_id: b1gv87ssvu49********
created_at: "2022-05-06T10:41:56Z"
...
network_settings:
  type: STANDARD
placement_policy: {}
```
For more information about the yc compute instance add-one-to-one-nat command, see the CLI reference.

Prepare a file with test data

Generate test data in URI format:
```
[Host: <internal_IP_address_of_test_target>]
[Connection: Close]
/ index
/test?param1=1&param2=2 get_test
```
Please note that the Connection: Close header means that each connection will be closed following the request. This mode is heavier on the application and load generator. If you do not want the connections closed, set Keep-Alive.

There are also two requests tagged index and get_test. The load generator will repeat them alternately within a given load profile.
Save the test data to a file named data.uri.

Run the test

In the management console, select Load Testing.
In the left-hand panel, select Tests. Click Create test.
In the Agents parameter, select agent-008.
Under Attached files, click Select files and select the data.uri file you saved earlier.
Under Test settings, select a configuration method: Form or Config.
Depending on the selected method, specify the test parameters:
Form

Config
1. In the Load generator field, select PANDORA.
2. In the Target address field, specify the internal IP address of the service to test.
3. In the Target port field, specify 443 (default HTTPS port). Allow using a secure connection.
4. In the Testing threads field, specify 5000.
  
  This means that the load generator can simultaneously process 5,000 operations: either create 5,000 connections or wait for 5,000 responses from the service at the same time.
  
  Tip
  
  For most tests, 1,000 to 10,000 threads are enough.
  
  More testing threads require more resources of the VM the agent is running on. Compute Cloud also has a limit of 50,000 of concurrent connections to a VM.
5. In the Load type menu, select RPS.
6. Click Load profile and enter the following description:
  - Profile 1: step
  - From: 1000
  - To: 5000
  - Step: 1000
  - Duration: 120s
  This is an instruction for the generator to increase the load from 1,000 to 5,000 requests per second in increments of 1,000 requests, 120 seconds each.
7. In the Request type field, select URI.
8. In the Attached files field, select Attached file.
9. In the Autostop menu, click Autostop and enter the following description:
  - Autostop type 1: QUANTILE
  - Quantile: 75
  - Response time limit: 100ms
  - Window duration: 10s
  This criterion stops the test if the 75th percentile exceeds 100 milliseconds for 10 seconds (for 10 seconds, the time to process 25% of requests exceeds 100 milliseconds).
10. Specify one more autostop:
  - Autostop type 2: INSTANCES
  - Limit: 90%
  - Window duration: 60s
  This criterion will stop the test if over 90% of the testing threads are busy for 60 seconds.
  
  As load increases, the system being tested will start to degrade at some point. Subsequent load increases will result in either an increased response time or an increased error rate. To avoid significantly increasing the test time, make sure to set Autostop as a termination criterion for these tests.
11. Under Forced test termination time, specify the time after which the test will autostop, unless it is stopped for other reasons. Set it to be slightly greater than the expected test duration.
12. Under Test information, specify the name, description, and number of the test version. This will make the report easier to read.
1. In the configuration input field, specify the testing thread settings in yaml format:
  pandora: enabled: true package: yandextank.plugins.Pandora config_content: pools: - id: HTTP gun: type: http # Protocol. target: <internal_IP_address_of_test_target>:443 ssl: true ammo: type: uri file: data.uri result: type: phout destination: ./phout.log rps: - duration: 120s # Test duration. type: step # Load type. from: 1000 to: 5000 step: 1000 startup: type: once times: 5000 # Number of testing threads. log: level: error monitoring: expvar: enabled: true port: 1234 autostop: # Autostop. enabled: true package: yandextank.plugins.Autostop autostop: - limit (5m) # Make sure to specify the time limit for the test. - quantile(75,100ms,10s) # Stop the test if, within 10 seconds, the 75 percentile # exceeds 100 milliseconds (within 10 seconds, the time # to process 25% of requests exceeds 100 milliseconds). - instances(90%,60s) # Stop the test if more than 90% of testing threads # get busy within 60 seconds. core: {} uploader: enabled: true package: yandextank.plugins.DataUploader job_name: '[example][pandora][step]' job_dsc: 'example' ver: '0.5.5' api_address: loadtesting.api.cloud.yandex.net:443
  As load increases, the system being tested will start to degrade at some point. Subsequent load increases will result in either an increased response time or an increased error rate. To avoid significantly increasing the test time, make sure to set Autostop as a termination criterion for these tests.
  
  Tip
  
  Check this sample configuration file. You can also find sample configuration files in existing tests.
Click Create.

Next, the configuration will be checked, and the agent will start loading the service.

Repeat the test

Once the test is complete, click Restart at the top right.
On the test creation page that opens, upload the same data.uri file you used to create the test.
Click Create.

Repeat these steps several times to get several completed tests.

Compare the results

To compare the results:

In the left-hand panel, select Tests and, in the tests table, select the completed tests you want to compare.
At the bottom of the pop-up panel, click Add to the comparison.
In the left-hand panel, select Comparison and view the combined charts of the load test results.
To compare results for a specific request, such as get_test, select the request from the Case drop-down list at the top of the page.
You can find a table with comparison results on the Tables page.

How to delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Comparing load test results

Get your cloud readyGet your cloud ready

Required paid resourcesRequired paid resources

Set up your infrastructureSet up your infrastructure

Create a service accountCreate a service account

Configure your networkConfigure your network

Configure security groupsConfigure security groups

Prepare a test targetPrepare a test target

Create a test agentCreate a test agent

Prepare a file with test dataPrepare a file with test data

Run the testRun the test

Repeat the testRepeat the test

Compare the resultsCompare the results

How to delete the resources you createdHow to delete the resources you created

Was the article helpful?