Creating test VMs via GitLab CI

You can use Yandex Cloud to automate routine tasks, such as running a specific script after each commit to the master branch in a Git repository. The example below creates and tests a VM following each commit.

To configure Continuous Integration (CI) for VM disk snapshots:

1. Create a VM for the test application: Create a new VM whose disk snapshot will be used to create new VMs via CI.
2. Set the VM up with a test application: Deploy a web server and components to support the test application to the VM. Write a test application that will reverse the words in a text sent to the server.
3. Check how the application works: Send a test request to check the server settings and the results.
4. Create a VM disk snapshot: Create a VM disk snapshot for CI to use to create new VMs.
5. Create a VM with GitLab: Create a VM with GitLab to store CI settings and a functionality test script in a repository.
6. Configure GitLab: Create a file repository and get the parameters required for configuration.
7. Configure Runner: Tool for performing tasks.
8. Configure CI: Set the CI configuration by specifying the necessary parameters for commands and testing.
9. Check that the application runs on the VM created using CI: Make sure that the VMs you are trying to create with CI and the VM snapshot have been created and the test application is functional.

If you no longer need the VMs you created, delete them.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Before creating a VM:

1. Go to the Yandex Cloud management console and select the folder where you will perform the operations.
2. Make sure the selected folder has a network with a subnet you can connect the VM to. To do this, select Virtual Private Cloud on the folder page. If the list contains a network, click its name to see the list of subnets. If there is neither network nor subnet, create them.

Required paid resourcesRequired paid resources

The infrastructure support costs include:

• Fee for continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for storing created images (see Compute Cloud pricing).
• Fee for using a dynamic public IP address (see Yandex Virtual Private Cloud pricing).

Create a VM for the test applicationCreate a VM for the test application

Create a VM to deploy the test application to, a set of components required for it to run, and a web server:

1. In the management console, select the folder to create your VM in.

2. In the list of services, select Compute Cloud.

3. In the left-hand panel, select Virtual machines.

4. Click Create virtual machine.

5. Under Boot disk image, select the Ubuntu 18.04 public image.

6. Under Location, select an availability zone to place your VM in.

7. Under Computing resources, navigate to the Custom tab and specify the parameters as follows:

   • Platform: Intel Ice Lake.
   • vCPU: 2.
   • Guaranteed vCPU performance: 20%
   • RAM: 1 GB.

8. In Network settings, select the subnet to connect the VM to once it is created.

9. Under Access, select SSH key and specify the information required to access the VM:

   • In the Login field, enter a name for the user you want to create on the VM, e.g., yc-user.

     Alert

     Do not use root or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use the sudo command.

   • In the SSH key field, select the SSH key saved in your organization user profile.

     If there are no saved SSH keys in your profile, or you want to add a new key:

     • Click Add key.
     • Enter a name for the SSH key.
     • Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
     • Click Add.

     The SSH key will be added to your organization user profile.

     If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.

10. Under General information, specify the VM name: ci-tutorial-test-app.

11. Click Create VM.

It may take a few minutes to create the VM. When its status changes to RUNNING, you can start to configure your VM.

Once created, the VM is assigned an IP address and a host name (FQDN). This data can be used for SSH access.

Prepare a VM with the test applicationPrepare a VM with the test application

On the created VM, deploy a collection of components required for the test application to run and a web server to handle requests. The application will be written in Python 2.

1. Under Network on the VM page in the management console, find the VM's public IP address.

2. Connect to the VM via SSH. To do this, use ssh in Linux or macOS, or PuTTY in Windows.

   ssh <login>@<VM_public_IP_address>
   

3. Run the apt update command to update the lists of packages available for installation.

4. Install the required packages: jq JSON processor, Git client, PIP package manager, virtualenv virtual environment management system, header files for the Python C API, and Nginx web server:

   sudo apt-get --yes install jq git python-pip virtualenv python-dev nginx-full
   

5. Create a directory to host the application. In addition, change the ownership of the directory to the user under which you will be connecting to the VM:

   sudo mkdir /srv/test-app
   sudo chown -R $USER /srv/test-app
   

6. Go to the directory and create the virtualenv virtual environment in it:

   cd /srv/test-app
   virtualenv test-venv
   

7. Activate the virtual environment:

   . test-venv/bin/activate
   

8. Install the Flask framework and the uWSGI web server in the virtual environment:

   pip install flask uwsgi
   

9. Deactivate the virtual environment:

   deactivate
   

10. In the /srv/test-app directory, create a file named api.py:

    touch api.py
    

11. Open the api.py file in any text editor and paste there the Python code which:

    • Accepts a text string as input in the text parameter.
    • Writes each word from the string in the reverse order.
    • Returns a response:
      • In JSON format if the client application can accept JSON.
      • In plain text if the client application does not accept JSON.

    # api.py
    import json
    from flask import Flask, request, make_response as response
    
    app = Flask(__name__)
    
    @app.route("/")
    def index():
        text = request.args.get('text')
        json_type = 'application/json'
        json_accepted = json_type in request.headers.get('Accept', '')
        if text:
            words = text.split()
            reversed_words = [word[::-1] for word in words]
            if json_accepted:
                res = response(json.dumps({'text': reversed_words}), 200)
            else:
                res = response(' '.join(reversed_words), 200)
        else:
            res = response('text not found', 501)
        res.headers['Content-Type'] = json_type if json_accepted else 'text/plain'
        return res
    

12. In the /srv/test-app directory, create a file named wsgi.py:

    touch wsgi.py
    

13. Open the wsgi.py file in any text editor and paste the code that runs the test application:

    # wsgi.py
    from api import app
    
    if __name__ == "__main__":
        app.run()
    

14. In the /srv/test-app directory, create a file named test-app.ini:

    touch test-app.ini
    

15. Open the test-app.ini file in any text editor and paste the configuration of the uWSGI server:

    #test-app.ini
    [uwsgi]
    module = wsgi:app
    
    master = true
    processes = 1
    
    socket = test-app.sock
    chmod-socket = 660
    vacuum = true
    
    die-on-term = true
    

16. Assign the www-data user as the owner of the /srv/test-app directory and the files in it:

    sudo chown -R www-data:www-data /srv/test-app
    

17. Prepare the service to start your uWSGI server. To do this, edit the /etc/systemd/system/test-app.service file as follows:

    #/etc/systemd/system/test-app.service
    [Unit]
    Description=uWSGI instance to serve test API
    After=network.target
    
    [Service]
    User=www-data
    Group=www-data
    WorkingDirectory=/srv/test-app
    Environment="PATH=/srv/test-app/test-venv/bin"
    ExecStart=/srv/test-app/test-venv/bin/uwsgi --ini test-app.ini
    
    [Install]
    WantedBy=multi-user.target
    

18. Provide the settings of the new virtual server in the Nginx configuration by changing the /etc/nginx/sites-available/test-app.conf file as follows:

    #/etc/nginx/sites-available/test-app.conf
    server {
        #server_name test-app.yandex www.test-app.yandex;
    
        listen 80;
    
        location /test/ {
            include uwsgi_params;
            uwsgi_param SCRIPT_NAME /test;
            uwsgi_modifier1 30;
            uwsgi_pass unix:/srv/test-app/test-app.sock;
        }
    }
    

19. Create a symbolic link that points to the test-app.conf Nginx configuration file:

    sudo ln -s /etc/nginx/sites-available/test-app.conf /etc/nginx/sites-enabled/
    

20. Delete the symbolic link that points to the default Nginx configuration:

    sudo unlink /etc/nginx/sites-enabled/default
    

21. Add the service to the system autorun list:

    sudo systemctl enable test-app.service
    

Check that the test application is workingCheck that the test application is working

To make sure the test application is functional and the web server is properly configured, send a simple request.

1. In the browser bar, enter the URL for testing the web server and application:

   http://<VM_public_IP_address>/test/?text=hello_world
   

2. If everything works correctly, the screen will display text with reversed words from the text parameter.

Take a snapshot of the VM diskTake a snapshot of the VM disk

To easily transfer the app and the web server configuration you created to VMs spawned with CI, you will need to take a snapshot of the test VM disk.

1. In the Yandex Cloud management console, select the folder you created your VM in.
2. Select Compute Cloud.
3. Find the ci-tutorial-test-app VM and select it.
4. Click Stop.
5. In the window that opens, click Stop.
6. After stopping the VM, select the Disks tab.
7. In the required disk row, click and select Create snapshot.
8. In the window that opens, enter a name for the snapshot: test-app-snap.
9. Click Create.

Create a VM with GitLabCreate a VM with GitLab

You can set up CI in Yandex Cloud by using a public image with GitLab pre-installed. GitLab includes a set of tools for managing Git repositories and configuring CI.

1. In the management console, select the folder to create your VM in.

2. In the list of services, select Compute Cloud.

3. In the left-hand panel, select Virtual machines.

4. Click Create virtual machine.

5. Under Boot disk image, go to the Marketplace tab, click Show all Marketplace products, and select the GitLab image.

6. Under Location, select an availability zone to place your VM in.

7. Under Computing resources, navigate to the Custom tab and specify the parameters as follows:

   • Platform: Intel Ice Lake.
   • vCPU: 2.
   • Guaranteed vCPU performance: 100%.
   • RAM: 2 GB.

8. In Network settings, select the subnet to connect the VM to once it is created.

9. Under Access, select SSH key and specify the information required to access the VM:

   • In the Login field, enter a name for the user you want to create on the VM, e.g., yc-user.

     Alert

     Do not use root or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use the sudo command.

   • In the SSH key field, select the SSH key saved in your organization user profile.

     If there are no saved SSH keys in your profile, or you want to add a new key:

     • Click Add key.
     • Enter a name for the SSH key.
     • Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
     • Click Add.

     The SSH key will be added to your organization user profile.

     If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.

10. Under General information, specify the VM name: ci-tutorial-gitlab.

11. Click Create VM.

It may take a few minutes to create the VM. When its status changes to RUNNING, you can start to configure your VM.

Once created, the VM is assigned an IP address and a host name (FQDN). This data can be used for SSH access.

Configure GitLabConfigure GitLab

To set GitLab up and configure the CI process, create a new project and enter the CI login credentials:

1. On the Compute Cloud page, select the created VM and copy its public IP.

2. Connect to the VM via SSH.

3. Get the GitLab administrator password using the following VM command:

   sudo cat /etc/gitlab/initial_root_password
   

4. Copy the password (without spaces) from the Password row to the clipboard or a separate file.

5. Open http://<VM_public_IP_address> in your browser. This will take you to the GitLab web interface.

6. Log in using the administrator account:

   • Username or email: root.
   • Password: Password you copied earlier.

   If you are unable to log in, reset the administrator account password.

7. Change the administrator account password.

8. Log in to the system again using the administrator account and the new password.

9. Select Create a project.

10. Set the project name: gitlab-test.

11. Click Create project.

12. Obtain an OAuth token from the Yandex OAuth service. To do this, follow the link and click Allow.

13. In the browser, open the link formatted as http://<VM_public_IP_address>/root.

14. Select the gitlab-test project.

15. On the screen that opens, choose the Settings tab on the left. In the menu that opens, select CI/CD.

16. Under Variables, click Expand.

17. Create a new variable:

    • Specify YC_OAUTH as the variable name.
    • Specify the OAuth token you received as the value of the variable.
    • Click Save variables.

18. Under Runners, click Expand.

19. Under Set up a specific Runner automatically, you will see the server address to connect to and the token for registering the server in the project. Use these values when registering the Runner.

Configure a RunnerConfigure a Runner

A Runner is a tool for performing tasks that a user creates. You need to install a Runner on the VM and register it in GitLab. For the Runner to be able to perform tasks, set up additional components: install the Yandex Cloud CLI and create a test to check the created VM.

1. Connect to the VM with GitLab over SSH:

   ssh <login>@<public_IP_address_of_VM_with_GitLab>
   

2. Add the new repository to the package manager:

   curl --location https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
   

3. Install GitLab Runner that will run your CI scripts:

   sudo apt-get -y install gitlab-runner
   

4. Register GitLab Runner:

   • For Please enter the gitlab-ci coordinator URL, enter the IP address of your GitLab server.
   • For Please enter the gitlab-ci token for this runner, enter your GitLab Runner token.
   • For Please enter the gitlab-ci description for this runner, enter the gitlab test runner description.
   • For Please enter the gitlab-ci tags for this runner, do not type anything. Press Enter.
   • For Please enter the executor, enter shell.

   sudo gitlab-runner register
   Runtime platform                                    arch=amd64 os=linux pid=8197 revision=3afdaba6 version=11.5.0
   Running in system-mode.
   
   Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
   http://<GitLab>_CI_IP_address/
   Please enter the gitlab-ci token for this runner:
   <Runner_token>
   Please enter the gitlab-ci description for this runner:
   [ci-tutorial-gitlab]: gitlab test runner
   Please enter the gitlab-ci tags for this runner (comma separated):
   
   Registering runner... succeeded                     runner=wZqzyy9s
   Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, shell, ssh, parallels:
   shell
   Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
   

5. For the CI script to be able to create VMs, please install the Yandex Cloud CLI:

   curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh --output install.sh
   sudo bash install.sh -n -i /opt/yc
   

6. To perform functional testing, install the pytest package:

   sudo apt-get install python-pytest
   

7. Create a file named test.py with the functional testing script:

   1. Open the home page of the gitlab-test repository.

   2. Click + and select New file.

   3. In the window that opens, name the file as test.py.

   4. Copy the following code to the file body:

      # test.py
      import pytest
      import json
      import socket as s
      
      @pytest.fixture
      def hostname(request):
          with open("instance-creation.out", "r") as fd:
              fqdn = json.loads(fd.read()).get("fqdn")
      
          return fqdn
      
      @pytest.fixture
      def socket(request):
          _socket = s.socket(s.AF_INET, s.SOCK_STREAM)
          def socket_teardown():
              _socket.close()
          request.addfinalizer(socket_teardown)
          return _socket
      
      def test_server_connect(socket, hostname):
          socket.connect((hostname, 80))
          assert socket
      

   5. Write any commit message and click Commit changes.

Configure CIConfigure CI

You need to define the configuration for CI.

1. Open the home page of the gitlab-test repository:

   http://<public_IP_address_of_GitLab_VM>/root/gitlab-test
   

2. Click Set up CI/CD. A window will open for adding a new file.

3. GitLab will name the file as .gitlab-ci.yml automatically. Do not rename it. Copy the following configuration to the file:

   #.gitlab-ci.yml
   stages:
     - build
     - test
   
   build:
     stage: build
     variables:
       snapshot_name: test-app-snap
       folder_id: <folder_ID>
       subnet_name: <subnet_name>
     script:
       - export instance_name="ci-tutorial-test-app-$(date +%s)"
       - export PATH="/opt/yc/bin:${PATH}"
       - yc config set token $YC_OAUTH
       - yc compute instance create
         --format json
         --name $instance_name
         --folder-id $folder_id
         --zone ru-central1-d
         --network-interface subnet-name=$subnet_name,nat-ip-version=ipv4
         --create-boot-disk name=$instance_name-boot,type=network-ssd,size=15,snapshot-name=$snapshot_name,auto-delete=true
         --memory 1
         --cores 1
         --hostname $instance_name > instance-creation.out
       - sleep 30
     artifacts:
       when: on_success
       paths:
       - instance-creation.out
       expire_in: 10 mins
   
   test_external:
     stage: test
     script:
       - py.test test.py > pytest-external.out
     artifacts:
       paths:
       - pytest-external.out
       expire_in: 1 hour
   

4. In the snapshot_name field, enter the snapshot name for the first VM.
   In the folder_id field, specify the ID of the folder where VMs are created.
   In the subnet_name field, specify the name of the subnet the VMs will connect to. You can get the name in the management console: open the appropriate folder and go to the Virtual Private Cloud service page.

5. Click Commit changes.

Test the application on the VM created using CITest the application on the VM created using CI

After making a commit, you need to make sure that CI worked correctly. The appropriate folder should contain a new VM where the test application and the web server are deployed.

To test the created VM:

1. Open the Yandex Cloud management console.

2. In the folder where the VMs were created, select Compute Cloud.

3. If everything is properly configured, the VM list should display a new VM, such as ci-tutorial-test-app-1543910277.

4. Select the created VM and copy its public IP address.

5. In the browser, open the following link:

   http://<public_IP_address_of_created_VM>/test/?text=hello_world
   

6. The application you created in the previous steps should also function on the created VM and return the reversed words from the text parameter.

Delete the resources you createdDelete the resources you created

If you no longer need the VMs and images:

• Delete the created VMs.
• Delete the created images.