Comparing load test results
In this use case, you will set up a load testing environment, run the tests, and compare the results of the load tests.
The use case uses the Pandora load generator and a simple web service as the test target.
To perform load testing and compare test results:
- Prepare your cloud.
- Prepare your infrastructure.
- Prepare a test target.
- Create an agent.
- Prepare a file with test data.
- Run a test.
- Run the same test several times.
- Compare the results.
If you no longer need the resources you created, delete them.
Prepare your cloud
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
If the agent is hosted on Yandex Cloud, you pay for computing resources (see Yandex Compute Cloud pricing).
At the Preview stage, Load Testing is free of charge.
Prepare the infrastructure
Create a service account
- Create a service account named
sa-loadtest
in the folder to host the agent that will generate the load. - Assign the following roles to the service account:
loadtesting.generatorClient
: Allows running the agent, performing tests on the agent, and uploading the results to storage.compute.admin
: Allows managing VMs in Compute Cloud.vpc.user
: Allows connecting to Virtual Private Cloud network resources and using them.
Configure a network
Create and configure a NAT gateway in the subnet where your test target and the agent will be placed. Thus, the agent will have access to Load Testing.
Configure security groups
-
Set up the test agent's security group:
- Create an agent security group named
agent-sg
. - Add rules:
-
Rule for outgoing HTTPS traffic to the Load Testing public API:
- Port range:
443
- Protocol:
TCP
- Destination name:
CIDR
- CIDR blocks:
0.0.0.0/0
This will allow connecting the agent to Load Testing to manage tests from the interface and get test results.
- Port range:
-
Rule for incoming SSH traffic:
- Port range:
22
- Protocol:
TCP
- Destination name:
CIDR
- CIDR blocks:
0.0.0.0/0
This will allow you to connect to the agent over SSH and manage tests from the console or collect debugging information.
- Port range:
-
Rule for outgoing traffic when generating load to the test target:
- Port range:
0-65535
- Protocol:
Any
- Destination name:
Security group
SelectFrom list
. Specify the security group where the test target is located.
Create this rule for each test target with a unique security group.
- Port range:
-
- Create an agent security group named
-
Set up the test target's security group:
-
Create the test target's security group named
load-target-sg
. -
Add a rule for incoming traffic when generating load to the test target:
- Port range:
0-65535
. - Protocol:
Any
. - Select
From list
. Specify the security group where the test target is located.
This rule allows agents to generate load to this target or enable additional monitoring tools.
- Port range:
-
Prepare a test target
To prepare a test target:
-
Create a VM from a public Linux image:
- Specify
load-target
as the name. - Specify, e.g.,
ycuser
as the username. - Specify the
load-target-sg
security group in the network settings section.
- Specify
-
Connect to the VM via SSH by running the command:
ssh ycuser@<VM_public_IP_address>
-
Create a web service file by running:
mkdir app cd app nano app.py
-
Paste the following code into the web service file:
from flask import Flask, make_response from flask_cors import CORS import os app = Flask(__name__) CORS(app) @app.route('/') def index(): res = make_response({"message":"Root URL"}, status_code=200) return res @app.route('/test') def get_test(): res = make_response({"message":"Get test URL"}, status_code=200) return res # Run application if __name__ == '__main__': app.run(host='0.0.0.0', debug=False, port=443)
-
Install the required packages:
pip install flask flask-cors
-
Run the service:
python3 app.py
The service will expect requests on port 443. Make sure the port is open and configured for HTTPS requests.
Create a test agent
-
If you do not have an SSH key pair yet, create one.
-
Create an agent.
Management console-
In the management console
, select the folder where you want to create your agent. -
In the list of services, select Load Testing.
-
In the Agents tab, click Create agent.
-
Give your agent a name, e.g.,
agent-008
. -
Specify the same availability zone where the test target is located.
-
Under Agent:
- Select the appropriate agent type. For more information, see Agent performance.
- Specify the subnet where the test target is located. Make sure you created and set up a NAT gateway in the subnet.
- If security groups are available to you, select a security group preset for the agent.
-
Under Access, specify the information required to access the agent:
-
Select the
sa-loadtest
service account. -
Enter the username in the Login field.
Alert
Do not use the
root
username or other names reserved by the operating system. To perform operations that require superuser permissions, use thesudo
command. -
In the SSH key field, paste the contents of the public key file.
-
-
Click Create.
-
Wait for the VM instance to create. Make sure the agent status changes to
Ready for test
.Note
The agent creation process may stop at the
Initializing connection
status unless the following conditions are met:- The agent has access to
loadtesting.api.cloud.yandex.net:443
and is assigned a public IP address. - A NAT gateway is set up in the target subnet.
- The service account assigned to the agent has the required roles.
- The agent has access to
-
-
Assign a public IP to the agent to enable access over SSH:
Management console- In the management console
, select the folder where the agent is located. - Select Compute Cloud.
- Select the VM named
agent-008
. - Under Network interface, in the top-right corner, click
and select Add public IP address. - In the window that opens:
- In the Public address field, select obtaining a Auto address.
- Click Add.
- In the management console
Prepare a file with test data
-
Generate payloads in URI format:
[Host: <test_target_internal_IP_address>] [Connection: Close] / index /test?param1=1¶m2=2 get_test
Please note that the
Connection: Close
header means each connection is terminated after making a request. This mode is heavier on the application and load generator. If you do not need to close connections, setKeep-Alive
.There are also two requests tagged
index
andget_test
. The load generator will repeat them within a given load profile. -
Save the payloads to a file named
data.uri
.
Run a test
-
In the management console
, select Load Testing. -
In the left-hand panel, select
Tests. Click Create test. -
In the Agents parameter, select
agent-008
. -
Under Attached files, click Select files and select the
data.uri
file you saved earlier. -
Under Test settings, select a configuration method: Form or Config.
-
Depending on the selected method, specify the test parameters:
FormConfig-
In the Load generator field, select PANDORA.
-
In the Target address field, specify the internal IP address of the service to test.
-
In the Target port field, specify
443
(default HTTPS port). Allow using a secure connection. -
In the Testing threads field, specify
5000
.This means that the load generator can simultaneously process 5,000 operations: either create 5,000 connections or wait for 5,000 responses from the service at the same time.
-
In the Load type menu, select
RPS
. -
Click
Load profile and enter the following description:- Profile 1:
Step
- From:
1000
- To:
5000
- Step:
1000
- Duration:
120s
This is an instruction for the generator to increase the load from 1,000 to 5,000 requests per second in increments of 1,000 requests, 120 seconds each.
- Profile 1:
-
In the Request type field, select
URI
. -
In the Attached files field, select Attached file.
-
In the Autostop menu, click
Autostop and enter the following description:- Autostop type 1:
QUANTILE
- Quantile:
75
- Response time limit:
100ms
- Window duration:
10s
This criterion stops the test if the 75th percentile exceeds 100 milliseconds for 10 seconds (for 10 seconds, the processing time of 25% of queries exceeds 100 milliseconds).
- Autostop type 1:
-
Specify one more autostop:
- Autostop type 2:
INSTANCES
- Limit:
90%
- Window duration:
60s
This criterion will stop the test if over 90% of the testing threads are busy for 60 seconds.
As load increases, the system being tested will start to degrade at some point. Subsequent load increases will result in either an increased response time or an increased error rate. To avoid significantly increasing the test time, make sure to set Autostop as a termination criterion for these tests.
- Autostop type 2:
-
Under Forced test termination time, specify the time to autostop the test unless it is stopped for other reasons. The parameter value should be slightly greater than the expected duration of the test.
-
Under Test information, specify the name, description, and number of the test version. This will make the report easier to read.
-
In the configuration input field, specify the testing thread settings in
yaml
format:pandora: enabled: true package: yandextank.plugins.Pandora config_content: pools: - id: HTTP gun: type: http # Protocol. target: <test_target_internal_IP_address>:443 ssl: true ammo: type: uri file: data.uri result: type: phout destination: ./phout.log rps: - duration: 120s # Test time. type: step # Load type. from: 1000 to: 5000 step: 1000 startup: type: once times: 5000 # Number of test threads. log: level: error monitoring: expvar: enabled: true port: 1234 autostop: # Autostop. enabled: true package: yandextank.plugins.Autostop autostop: - limit (5m) # Make sure to specify the time limit for the test. - quantile(75,100ms,10s) # Stop the test if the 75th percentile # exceeds 100 milliseconds in 10 seconds (within 10 seconds, # the time to process 25% of requests exceeds 100 milliseconds). - instances(90%,60s) # Stop the test if 90% of test threads # get busy within 60 seconds. core: {} uploader: enabled: true package: yandextank.plugins.DataUploader job_name: '[example][pandora][step]' job_dsc: 'example' ver: '0.5.5' api_address: loadtesting.api.cloud.yandex.net:443
As load increases, the system being tested will start to degrade at some point. Subsequent load increases will result in either an increased response time or an increased error rate. To avoid significantly increasing the test time, make sure to set Autostop as a termination criterion for these tests.
Tip
View a sample configuration file. You can also find sample configuration files in existing tests.
-
-
Click Create.
Once you do that, the configuration will pass checks, and the agent will start loading the service you are testing.
Repeat the test
- When the test is complete, click
Restart at the top right. - On the test creation page that opens, upload the same
data.uri
file that you used to create the test. - Click Create.
Repeat these steps several times to get several completed tests.
Compare the results
To compare the results:
- In the left-hand panel, select
Tests, and in the test table, select the completed tests you want to compare. - At the bottom of the pop-up panel, click Add to the comparison.
- In the left-hand panel, select Comparison and view the combined charts of the load test results.
- To compare results for a specific request, such as
get_test
, select the required request from the Case drop-down list at the top of the page. - You can find a table with comparison results on the Tables page.
How to delete the resources you created
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need: