Creating test VMs via GitLab CI
You can use Yandex Cloud to automate routine tasks, such as running a specific script after each Git master
branch commit. In the following example, we create and test a VM after each commit.
To configure continuous integration (CI) for VM disk snapshots:
- Create the test application VM: Create a virtual machine allowing you to make new CI VMs using its disk snapshot.
- Configure the test application VM: Install a web server and test application components on the VM. Write a test application reversing words in a text sent to the server.
- Check how the application works: Send a test request to check whether the server settings are correct.
- Create a VM disk snapshot: Create a disk snapshot that CI will use to create new VMs.
- Create a GitLab VM: Create a GitLab
VM with a repository to store CI settings and a test script. - Configure GitLab: Create a file repository and get configuration parameters.
- Configure Runner: Tool for performing tasks.
- Configure CI: Specify the required command and testing parameters.
- Make sure the application runs on the CI VM: Check whether the system created CI snapshot VMs and they run your test application.
If you no longer need the VMs you created, delete them.
Get your cloud ready
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Before creating a VM:
- Navigate to the Yandex Cloud management console
and select the folder to work with. - Make sure the selected folder has a network with a subnet where you can connect your VM. To do this, select Virtual Private Cloud on the folder page. If the list contains a network, click its name to see the list of subnets. If there are no networks in the list or the selected network has no subnets, create them as required.
Required paid resources
The infrastructure support costs include:
- Fee for continuously running VMs (see Yandex Compute Cloud pricing).
- Fee for storing created images (see Compute Cloud pricing).
- Fee for a dynamic public IP address (see Yandex Virtual Private Cloud pricing).
Create the test application VM
Create a VM where you will install a web server, your test application, and its components:
-
In the management console
, select the folder where you want to create your VM. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Under Boot disk image, select the Ubuntu 18.04 public image.
-
Under Location, select an availability zone where your VM will reside.
-
Under Computing resources, navigate to the Custom tab and specify these parameters:
- Platform:
Intel Ice Lake
- vCPU:
2
- Guaranteed vCPU performance:
20%
- RAM:
1 GB
- Platform:
-
In Network settings, select the subnet where you will connect your VM.
-
Under Access, select SSH key and specify the VM access credentials:
-
In the Login field, specify the VM user name, e.g.,
yc-user
.Alert
Do not use
root
or other reserved usernames. To perform operations requiring root privileges, use thesudo
command. -
In the SSH key field, select the SSH key saved in your organization user profile.
If there are no saved SSH keys in your profile, or you want to add a new key:
- Click Add key.
- Enter a name for the SSH key.
- Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
- Click Add.
The SSH key will be added to your organization user profile.
If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.
-
-
Under General information, specify the VM name:
ci-tutorial-test-app
. -
Click Create VM.
It may take a few minutes to create your VM. When its status changes to RUNNING
, you can start configuring it.
Once your VM is created, the system will assign it an IP address and a host name (FQDN). You can use them for SSH access.
Configure the test application VM
On the new VM, install a web server and test application components. You will use Python 2 to create your test application.
-
In the management console
, under Network on the VM page, find the VM public IP address. -
Connect to the VM over SSH. You can do it using
ssh
in Linux or macOS or PuTTY in Windows.ssh <login>@<VM_public_IP_address>
-
Run the
apt update
command to update the lists of packages available for installation. -
Install the required packages: jq JSON processor, Git client, PIP package manager,
virtualenv
virtual environment management system, Python C API header files, and Nginx web server:sudo apt-get --yes install jq git python-pip virtualenv python-dev nginx-full
-
Create your application directory and make yourself its owner:
sudo mkdir /srv/test-app sudo chown -R $USER /srv/test-app
-
Navigate to this directory and create the
virtualenv
virtual environment in it:cd /srv/test-app virtualenv test-venv
-
Activate the virtual environment:
. test-venv/bin/activate
-
Install the Flask framework and the uWSGI web server in the virtual environment:
pip install flask uwsgi
-
Deactivate the virtual environment:
deactivate
-
Create the
api.py
file in the/srv/test-app
directory:touch api.py
-
Open the
api.py
file in any text editor and paste the Python code below into it; the code does the following:- Accepts a text string from the
text
input argument. - Reverses each word in the string while retaining spaces.
- Returns a response:
- In JSON format if the client accepts it.
- In plain text if the client does not accept JSON.
# api.py import json from flask import Flask, request, make_response as response app = Flask(__name__) @app.route("/") def index(): text = request.args.get('text') json_type = 'application/json' json_accepted = json_type in request.headers.get('Accept', '') if text: words = text.split() reversed_words = [word[::-1] for word in words] if json_accepted: res = response(json.dumps({'text': reversed_words}), 200) else: res = response(' '.join(reversed_words), 200) else: res = response('text not found', 501) res.headers['Content-Type'] = json_type if json_accepted else 'text/plain' return res
- Accepts a text string from the
-
Create the
wsgi.py
file in the/srv/test-app
directory:touch wsgi.py
-
Open the
wsgi.py
file in any text editor and paste the application starting code in it:# wsgi.py from api import app if __name__ == "__main__": app.run()
-
Create the
test-app.ini
file in the/srv/test-app
directory:touch test-app.ini
-
Open the
test-app.ini
file in any text editor and paste the uWSGI server configuration in it:#test-app.ini [uwsgi] module = wsgi:app master = true processes = 1 socket = test-app.sock chmod-socket = 660 vacuum = true die-on-term = true
-
Make the
www-data
user owner of the/srv/test-app
directory and its contents:sudo chown -R www-data:www-data /srv/test-app
-
Prepare the service that will start your uWSGI server. To do this, edit the
/etc/systemd/system/test-app.service
file as follows:#/etc/systemd/system/test-app.service [Unit] Description=uWSGI instance to serve test API After=network.target [Service] User=www-data Group=www-data WorkingDirectory=/srv/test-app Environment="PATH=/srv/test-app/test-venv/bin" ExecStart=/srv/test-app/test-venv/bin/uwsgi --ini test-app.ini [Install] WantedBy=multi-user.target
-
Specify the new virtual server settings in the Nginx configuration by changing the
/etc/nginx/sites-available/test-app.conf
file as follows:#/etc/nginx/sites-available/test-app.conf server { #server_name test-app.yandex www.test-app.yandex; listen 80; location /test/ { include uwsgi_params; uwsgi_param SCRIPT_NAME /test; uwsgi_modifier1 30; uwsgi_pass unix:/srv/test-app/test-app.sock; } }
-
Create a symbolic link pointing to the
test-app.conf
Nginx configuration file:sudo ln -s /etc/nginx/sites-available/test-app.conf /etc/nginx/sites-enabled/
-
Delete the symbolic link pointing to the default Nginx configuration:
sudo unlink /etc/nginx/sites-enabled/default
-
Configure the service to run at boot:
sudo systemctl enable test-app.service
Check whether the test application is working
To make sure the web server is properly configured and the test application is working as expected, make a simple request.
-
In the browser address bar, enter the following URL:
http://<VM_public_IP_address>/test/?text=hello_world
-
If everything works correctly, you will see reversed words from the
text
argument string.
Create a VM disk snapshot
Create a VM disk snapshot to transfer your web server configuration and test application to CI VMs.
- In the Yandex Cloud management console
, select the folder where you created your VM. - Select Compute Cloud.
- Find the
ci-tutorial-test-app
VM and select it. - Click Stop.
- In the window that opens, click Stop.
- Once the VM gets stopped, select the Disks tab.
- Select the disk you need, click
, and select Create snapshot. - In the window that opens, specify the snapshot name:
test-app-snap
. - Click Create.
Create a GitLab VM
You can set up CI in Yandex Cloud by using a public image with GitLab pre-installed. GitLab includes tools for managing Git repositories and configuring CI.
-
In the management console
, select the folder where you want to create your VM. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Under Boot disk image, navigate to the Marketplace tab, click Show all Marketplace products, and select the GitLab image.
-
Under Location, select an availability zone where your VM will reside.
-
Under Computing resources, navigate to the Custom tab and specify these parameters:
- Platform:
Intel Ice Lake
- vCPU:
2
- Guaranteed vCPU performance:
100%
- RAM:
2 GB
- Platform:
-
In Network settings, select the subnet where you will connect your VM.
-
Under Access, select SSH key and specify the VM access credentials:
-
In the Login field, specify the VM user name, e.g.,
yc-user
.Alert
Do not use
root
or other reserved usernames. To perform operations requiring root privileges, use thesudo
command. -
In the SSH key field, select the SSH key saved in your organization user profile.
If there are no saved SSH keys in your profile, or you want to add a new key:
- Click Add key.
- Enter a name for the SSH key.
- Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
- Click Add.
The SSH key will be added to your organization user profile.
If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.
-
-
Under General information, specify the VM name:
ci-tutorial-gitlab
. -
Click Create VM.
It may take a few minutes to create your VM. When its status changes to RUNNING
, you can start configuring it.
Once your VM is created, the system will assign it an IP address and a host name (FQDN). You can use them for SSH access.
Configure GitLab
To set GitLab up and configure the CI process, create a new project and enter your CI login credentials:
-
On the Compute Cloud page, select the created VM and copy its public IP address.
-
Connect to the VM over SSH.
-
Get the GitLab administrator password with the following VM command:
sudo cat /etc/gitlab/initial_root_password
-
Copy the password without spaces from the
Password
line to the clipboard or a separate file. -
In your browser, open
http://<VM_public_IP_address>
. This will take you to the GitLab web interface. -
Log in as the administrator:
- Username or email:
root
. - Password: Password you copied in the previous step.
If you are unable to log in, reset the administrator password
. - Username or email:
-
Log in as the administrator with the new password.
-
Select Create a project.
-
Specify the project name:
gitlab-test
. -
Click Create project.
-
Get a Yandex OAuth token. To do this, follow this link
and click Allow. -
In your browser, open this link:
http://<VM_public_IP_address>/root
. -
Select the
gitlab-test
project. -
On the screen that opens, click Settings on the left and, in the popup menu, select CI/CD.
-
Under Variables, click Expand.
-
Create a new variable:
- Specify the variable name:
YC_OAUTH
. - As the value of the variable, specify the OAuth token you received previously.
- Click Save variables.
- Specify the variable name:
-
Under Runners, click Expand.
-
Under Set up a specific Runner automatically, you will see the GitLab server address and the Runner token. You will need them when registering your runner.
Configure your runner
Runner is a tool for running user tasks. You need to create a runner on your VM and register it in GitLab. To make your runner work, install the Yandex Cloud CLI and create a test to check the new VM.
-
Connect to the GitLab VM over SSH:
ssh <login>@<public_IP_address_of_VM_with_GitLab>
-
Add a new repository to the package manager:
curl --location https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
-
Install GitLab Runner to run your CI scripts:
sudo apt-get -y install gitlab-runner
-
Register your runner:
- In response to
Please enter the gitlab-ci coordinator URL
, specify your GitLab IP address. - In response to
Please enter the gitlab-ci token for this runner
, speciffy your GitLab Runner token. - In response to
Please enter the gitlab-ci description for this runner
, specify thegitlab test runner
description. - In response to
Please enter the gitlab-ci tags for this runner
, press Enter. - In response to
Please enter the executor
, specifyshell
.
sudo gitlab-runner register Runtime platform arch=amd64 os=linux pid=8197 revision=3afdaba6 version=11.5.0 Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): http://<GitLab>_CI_IP_address/ Please enter the gitlab-ci token for this runner: <Runner_token> Please enter the gitlab-ci description for this runner: [ci-tutorial-gitlab]: gitlab test runner Please enter the gitlab-ci tags for this runner (comma separated): Registering runner... succeeded runner=wZqzyy9s Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, shell, ssh, parallels: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
- In response to
-
Install the Yandex Cloud CLI so you will be able to create VMs with the CI script:
curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh --output install.sh sudo bash install.sh -n -i /opt/yc
-
Install the
pytest
package for functional testing:sudo apt-get install python-pytest
-
Create the
test.py
file with the functional testing script:-
Open the
gitlab-test
repository home page. -
Click + and select New file.
-
In the window that opens, specify the file name:
test.py
. -
Add the following code to the file:
# test.py import pytest import json import socket as s @pytest.fixture def hostname(request): with open("instance-creation.out", "r") as fd: fqdn = json.loads(fd.read()).get("fqdn") return fqdn @pytest.fixture def socket(request): _socket = s.socket(s.AF_INET, s.SOCK_STREAM) def socket_teardown(): _socket.close() request.addfinalizer(socket_teardown) return _socket def test_server_connect(socket, hostname): socket.connect((hostname, 80)) assert socket
-
Specify your commit message and click Commit changes.
-
Configure CI
Now you need to configure CI.
-
Open the
gitlab-test
repository home page:http://<public_IP_address_of_GitLab_VM>/root/gitlab-test
-
Click Set up CI/CD. This will open a window for adding a new file.
-
GitLab will automatically name the file as
.gitlab-ci.yml
. Do not rename it. Add the following configuration to the file:#.gitlab-ci.yml stages: - build - test build: stage: build variables: snapshot_name: test-app-snap folder_id: <folder_ID> subnet_name: <subnet_name> script: - export instance_name="ci-tutorial-test-app-$(date +%s)" - export PATH="/opt/yc/bin:${PATH}" - yc config set token $YC_OAUTH - yc compute instance create --format json --name $instance_name --folder-id $folder_id --zone ru-central1-d --network-interface subnet-name=$subnet_name,nat-ip-version=ipv4 --create-boot-disk name=$instance_name-boot,type=network-ssd,size=15,snapshot-name=$snapshot_name,auto-delete=true --memory 1 --cores 1 --hostname $instance_name > instance-creation.out - sleep 30 artifacts: when: on_success paths: - instance-creation.out expire_in: 10 mins test_external: stage: test script: - py.test test.py > pytest-external.out artifacts: paths: - pytest-external.out expire_in: 1 hour
-
In the
snapshot_name
field, specify the first VM snapshot name.
In thefolder_id
field, specify the VM folder ID.
In thesubnet_name
field, specify your VM subnet name; you can find this name on the Virtual Private Cloud page in the relevant folder of the management console. -
Click Commit changes.
Test the application on the CI VM
After making a commit, make sure that CI performed its function correctly. You should see a new VM with a web server and your test application in the relevant folder..
To test the new VM:
-
Open the Yandex Cloud management console.
-
In the folder with the VMs, select Compute Cloud.
-
If you did everything correctly, you will see a new VM in the list, e.g.,
ci-tutorial-test-app-1543910277
. -
Select the new VM and copy its public IP address.
-
In your browser, open this link:
http://<public_IP_address_of_created_VM>/test/?text=hello_world
-
Your test application should also work on the new VM, returning reversed words from the
text
argument.
Delete the resources you created
If you no longer need the VMs and images you created: