Deploying a self-hosted SourceCraft worker on a Yandex Compute Cloud VM

Self-hosted workers are users’ personal servers, both virtual and physical, on which SourceCraft CI/CD processes run. These processes will have access to the user server environment.

Restrictions on the amount of computing resources do not apply to self-hosted workers, and they also do not consume the total CI/CD runtime quota.

If you have a computer with a suitable configuration, you can set up a self-hosted worker for SourceCraft all by yourself. Otherwise, you can create a VM in Compute Cloud and install a self-hosted worker on it. Even more, you can automate not only the processes of creating a VM, installing, configuring, and starting a self-hosted worker, but also the process of deleting a VM after the CI/CD workflow is completed. This way you can optimize the use of your Compute Cloud resources.

Here we discuss an example of how to implement such automation with the help of SourceCraft CI/CD processes using a service connection.

Self-hosted worker use cases:

• In the same repository the worker was created from. In cases like this, the VM can be automatically deleted as soon as the workflow is completed.

  test-task-and-delete-vm-async workflow example

  test-task-and-delete-vm-async:
    runs_on: self-hosted
    tasks:
      - test-task
      - name: delete
        uses: delete-vm-async
        needs: [test-task]
  

  The test-task task performs useful work, such as project compilation.

  The delete task gets started only after test-task (the needs: [test-task] parameter) is executed and asynchronously deletes the VM on which the self-hosted worker was run.

  Before executing the test-task-and-delete-vm-async workflow, you need to execute the create-vm workflow the self-hosted worker will be created in.

  Note

  You cannot set a precise workflow sequence, so workflows must be started manually one by one.

• In any other repository of the organization. In this case, you need to delete the VM with the self-hosted worker manually.

To deploy a self-hosted SourceCraft worker on a Compute Cloud VM:

1. Get ready.
2. Deploy the infrastructure.
3. Create a repository with a CI/CD configuration.
4. Create secrets.
5. Set up a service connection.
6. Set up a CI/CD process.
7. Test the worker.

If you no longer need the resources you created, delete them.

Get ready for workGet ready for work

1. Sign up for Yandex Cloud and create a billing account:

   1. Navigate to the management console and log in to Yandex Cloud or create a new account.
   2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

   If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

   Learn more about clouds and folders here.

2. Authenticate in SourceCraft on the service home page or sign up.

Required paid resourcesRequired paid resources

The cost of supporting a self-hosted worker infrastructure includes:

• Fee for a disk and a running VM (see Yandex Compute Cloud pricing).
• Fee for using an external IP address (see Yandex Virtual Private Cloud pricing).

Deploy the infrastructureDeploy the infrastructure

1. If you do not already have a network infrastructure in the Yandex Cloud folder you want to have your self-hosted worker VM in, create a cloud network, e.g., default, and a subnet, e.g., default-ru-central1-d.

   Note

   The example CI/CD configuration discussed here uses a network named default and a subnet named default-ru-central1-d.

2. In the given example, the default security group is used for the VM. Add the following rules to it:

   
   

   Traffic direction	Description	Port range	Protocol	Source / Destination name	CIDR blocks
   Inbound	Access to VM from the internet	0-65535	Any	CIDR	0.0.0.0/0
   Outbound	VM access to the internet	0-65535	Any	CIDR	0.0.0.0/0

3. In the Yandex Cloud folder you want to have your self-hosted worker VM in, create a service account with the compute.editor role for the folder.

   This service account will be used to create VMs in Compute Cloud and establish a SourceCraft service connection.

Create a repository with a CI/CD configurationCreate a repository with a CI/CD configuration

1. Create a new empty repository in the organization whose cloud and folder will be used for the self-hosted worker VM.

   On the page that opens, copy the link for access to the new repository formatted as: https://git@git.sourcecraft.dev/<organization_slug>/<new_repository_slug>.git.

   Note

   In the example below, you work with git via HTTPS. You can also use SSH.

2. Clone the self-hosted-worker repository with an example of a self-hosted worker deployment:

   git clone https://git@git.sourcecraft.dev/examples/self-hosted-worker.git
   cd self-hosted-worker
   

   The self-hosted-worker repository contains the following files:

   ├── .sourcecraft
   │   └── ci.yaml                      # SourceCraft CI/CD configuration
   ├── .gitignore
   ├── compute-instance-create.sh       # script for creating a VM
   ├── compute-instance-delete-async.sh # script for deleting a VM in asynchronous mode
   ├── compute-instance-delete.sh       # script for deleting a VM
   ├── metadata.yaml                    # metadata for a VM
   └── README.md
   

3. Use the self-hosted-worker repository as the basis for your new repository:

   git remote remove origin
   git remote add origin https://git@git.sourcecraft.dev/<organization_slug>/<new_repository_slug>.git
   git branch -M main
   

4. Push your changes to the new repository:

   git push -u origin main
   

Create secretsCreate secrets

1. Get a personal authentication token (PAT) with the Repository administrator role and access to the repository you want to be using the self-hosted CI/CD worker in.

   Tip

   If you want to use the worker for more than just the repository it was created from, specify the appropriate permissions for the personal token.

   The worker uses this token to get authenticated in SourceCraft.

2. Generate SSH keys for access to the worker VM.

3. Create the following SourceCraft secrets:

   • With the PAT name and the token value.
   • With the SSH_PUB name and a public SSH key.

Set up a service connectionSet up a service connection

Service connections allow you to securely integrate your SourceCraft projects with the Yandex Cloud resources.

Create a service connection named default-service-connection, use the previously created service account with the compute.editor role for the folder.

Set up a CI/CD processSet up a CI/CD process

Set up a CI/CD process to deploy a self-hosted worker.

Configure basic CI/CD settingsConfigure basic CI/CD settings

The CI/CD process is configured in the .sourcecraft/ci.yaml file.

The presented CI/CD configuration consists of the following workflows:

• create-vm: Creates a VM with a self-hosted worker.
• delete-vm: Deletes a VM with a self-hosted worker.
• test-task-and-delete-vm-async: Runs a test task on a self-hosted worker and deletes the VM asynchronously.

Adjust the CI/CD configuration for your needs or use an existing one.

Set your VM parametersSet your VM parameters

The VM configuration for a self-hosted worker is specified in the env variable section in the .sourcecraft/ci.yaml file:

• COMPUTE_INSTANCE_NAME: VM name and host name, e.g., self-hosted-worker.
• VPC_SUBNET_NAME: Name of the subnet created earlier the VM will reside in, e.g., default-ru-central1-d.
• CREATE_BOOT_DISK: Disk image from which the VM will be created, e.g., image-folder-id=standard-images,image-family=ubuntu-2204-lts (Ubuntu 22.04 LTS). See also Getting a list of public images.
• PLATFORM: VM platform, e.g., standard-v3.
• MEMORY: Amount of RAM, e.g., 64GB.
• CORES: Number of vCPU cores, e.g., 8.
• CORE_FRACTION: Guaranteed share of vCPUs, e.g., 100.
• PREEMPTIBLE: Whether or not to use a preemptible VM, e.g., false.
• SSH_PUB: Previously created secret with the public part of the SSH key in ${{ secrets.<secret_name> }} format. Set in case you need to connect to a VM with a self-hosted worker.

A VM user with the builder login is created within the metadata.yaml file.

users:
- name: builder
  sudo: 'ALL=(ALL) NOPASSWD:ALL'
  shell: /bin/bash
  ssh_authorized_keys:
  - ${SSH_PUB}

Adjust the VM configuration for your needs or use an existing one.

Create a script to install a self-hosted worker and other software on the VMCreate a script to install a self-hosted worker and other software on the VM

To install an executable of a self-hosted worker or other software, e.g., Docker, the metadata.yaml file is used:

  - path: "/usr/local/etc/startup.sh"
    permissions: "755"
    content: |
      #!/bin/bash

      # install self-hosted-processor
      apt-get update
      apt-get install -y musl
      curl \
        --silent \
        --show-error \
        --location \
        https://storage.yandexcloud.net/src-processor-downloads/self-hosted-processor-latest/linux/amd64/self-hosted-processor \
        --output self-hosted-processor
      chmod +x self-hosted-processor
      mv self-hosted-processor /home/builder/sourcecraft
      chmod 777 -R /home/builder/sourcecraft

      # install docker
      apt install ca-certificates curl software-properties-common apt-transport-https -y
      install -m 0755 -d /etc/apt/keyrings
      curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
      apt update
      apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
      usermod -aG docker builder

In a newly deployed VM, the /usr/local/etc/startup.sh bash script is created and executed, which installs the self-hosted worker executable and Docker.

Adjust the installation script for your needs or use an existing one.

Configure a self-hosted workerConfigure a self-hosted worker

The config.yaml worker configuration file is also configured in the metadata.yaml file:

write_files:
  - path: "/home/builder/sourcecraft/config.yaml"
    permissions: "644"
    content: |
      executor_type: shell
      self_hosted:
        root_dir: /home/builder/sourcecraft/tmp
        max_slots: 3
      logger_type: json
      logger_level: info
      endpoint:
        host: ci.sourcecraft.tech
        port: 443
        ssl_no_verify: false
      auth:
        pat: ${PAT}

Test the workerTest the worker

Create a VM with a self-hosted workerCreate a VM with a self-hosted worker

Manually run the create-vm workflow.

You can follow the creation process using the cube logs. If the workflow completes successfully, you can see the new VM in the Yandex Cloud management console using the link from the create-vm cube log.

Run a test workflowRun a test workflow

1. Manually run the test-task-and-delete-vm-async workflow.
2. In test-task, check the logs of the test-worker and test-docker cubes:
   • The test-worker cube logs should present the results of the uname -a and free -h commands run directly on the worker.
   • The test-docker cube logs should present the version of the Yandex Cloud CLI that will be run in the Docker container.

After test-task is completed, the delete (delete-vm-async) task will be started, which will attempt to delete the VM in asynchronous mode. You can see the deletion result in the Yandex Cloud management console using the link from the delete-vm cube log.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created, manually run the delete-vm workflow or delete the self-hosted worker VM in the Yandex Cloud interface.

Optionally, delete the network and subnet.

See alsoSee also

• Setting up a self-hosted worker for SourceCraft
• Creating a VM with a custom configuration script
• Getting a list of public images
• Platforms
• vCPU performance levels