Automating image builds using Jenkins and Packer

Based on the specified configuration, Packer creates VM disk images in Yandex Compute Cloud. Jenkins enables continuous integration (CI).

You can use images to create a cloud infrastructure using such IaC tools as Terraform.

To install and configure Jenkins, Packer, GitHub, and Terraform to work together:

1. Get your cloud ready.
2. Configure the environment.
3. Create a service account.
4. Create a Jenkins VM.
5. Install Packer to your VM.
6. Configure Jenkins.
7. Set up a Jenkins task.
8. Configure the GitHub repository.
9. Create an image using Jenkins.
10. Deploy the images using Terraform.

If you no longer need the VM or the cluster you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The infrastructure support costs include:

• Fee for continuously running VMs (see Compute Cloud pricing).
• Fee for storing created images (see Compute Cloud pricing).
• Fee for using a dynamic public IP address (see Yandex Virtual Private Cloud pricing).

Configure the environmentConfigure the environment

Set up the software:

• Install the Yandex Cloud command line interface.
• Install Terraform. See also: Getting started with Terraform.
• Download the jq utility.
• Configure Git. If you are running Windows, use Git Bash.
• Create a repository branch with examples in your GitHub account.
• Prepare an SSH key for VM access.

Create a service accountCreate a service account

Jenkins uses service accounts to perform actions in your cloud and folder. To create a service account:

1. Get the folder and cloud IDs by running yc config list.

2. Create a service account and provide its ID to the environment variable using these commands:

   yc iam service-account create --name <username>
   yc iam key create --service-account-name <username> -o <username.json>
   SERVICE_ACCOUNT_ID=$(yc iam service-account get --name <username> --format json | jq -r .id)
   

   This will create a JSON file containing login credentials in the current folder.

3. Assign the service account the admin role for the folder where the operations will be performed:

   yc resource-manager folder add-access-binding <folder_name> --role admin --subject serviceAccount:$SERVICE_ACCOUNT_ID
   

Create a Jenkins VMCreate a Jenkins VM

Jenkins will get VM image configuration changes from GitHub and then use Packer to create images in the cloud.

To create a VM:

1. In the management console, select the folder where you want to create your VM.

2. From the list of services, select Compute Cloud.

3. In the left-hand panel, select Virtual machines.

4. Click Create virtual machine.

5. Under Boot disk image, go to the Marketplace tab, click Show all Marketplace products, and select the Jenkins image.

   Note

   To configure the Jenkins VM manually, follow this guide.

6. Under Location, select an availability zone where your VM will reside.

7. Under Disks and file storages, enter 15 GB as your boot disk size.

8. Under Computing resources, navigate to the Custom tab and specify the parameters as follows:

   • Platform: Intel Ice Lake
   • vCPU: 2
   • Guaranteed vCPU performance: 20%
   • RAM: 2 GB

9. Under Network settings:

   • In the Subnet field, enter the ID of a subnet in the new VM’s availability zone. Alternatively, select a cloud network from the list.

     • Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.

     • If you do not have a network, click Create network to create one:

       • In the window that opens, specify the network name and select the folder to host the network.
       • Optionally, enable the Create subnets setting to automatically create subnets in all availability zones.
       • Click Create network.

   • In the Public IP address field, select Auto to assign a random external IP address from the Yandex Cloud pool to the VM. Alternatively, select a static address from the list if you reserved one.

10. Under Access, select SSH key and specify the VM access credentials:

    • In the Login field, enter a name for the user you want to create on the VM, e.g., yc-user.

      Alert

      Do not use root or other reserved usernames. To perform operations requiring root privileges, use the sudo command.

    • In the SSH key field, select the SSH key saved in your organization user profile.

      If there are no saved SSH keys in your profile, or you want to add a new key:

      • Click Add key.
      • Enter a name for the SSH key.
      • Upload or paste the contents of the public key file. You need to create a key pair for the SSH connection to a VM yourself.
      • Click Add.

      The SSH key will be added to your organization user profile.

      If users cannot add SSH keys to their profiles in the organization, the added public SSH key will only be saved to the user profile of the VM being created.

11. Under General information, specify the VM name: jenkins-tutorial.

12. Click Create VM.

Install PackerInstall Packer

Packer enables you to create VM disk images with parameters specified in a configuration file.

1. Download a Packer distribution kit for Linux. You can also download a Packer distribution kit for your platform from here.

2. Upload Packer to the VM you created:

   scp packer_<Packer_version>_linux_amd64.zip <login>@<VM_public_IP_address>:~/
   

3. Connect to the VM over SSH. To do this, use ssh in Linux or macOS, or PuTTY in Windows.

4. Create a new folder, move the Packer executables there, and unpack the archive:

   sudo mkdir /opt/yandex-packer/
   sudo mv packer_<Packer_version>_linux_amd64.zip /opt/yandex-packer/
   unzip packer_<Packer_version>_linux_amd64.zip
   

5. Configure the Yandex Compute Builder plugin:

   1. Create a file named config.pkr.hcl in the /opt/yandex-packer/ directory with the following contents:

      packer {
        required_plugins {
          yandex = {
            version = ">= 1.1.2"
            source  = "github.com/hashicorp/yandex"
          }
        }
      }
      

   2. Install the plugin:

      packer init <path_to_config.pkr.hcl>
      

      Result:

      Installed plugin github.com/hashicorp/yandex v1.1.2 in ...
      

6. Jenkins will run all its actions on behalf of thejenkins user. This is why you need to grant this user the permissions to launch Packer:

   sudo chmod u+x /opt/yandex-packer/packer*
   sudo chown jenkins:jenkins /opt/yandex-packer/packer*
   

Configure JenkinsConfigure Jenkins

To build images based on configurations from GitHub, configure Jenkins:

1. Connect to the VM over SSH using ssh in Linux or macOS or PuTTY in Windows.

2. Open the password file required to launch the configuration and copy the password:

   sudo cat /var/lib/jenkins/secrets/initialAdminPassword
   

3. In your browser, go to http://<public_IP_address_of_VM_with_Jenkins>. Open the Jenkins management console.

4. Enter the password you copied in the Administrator password field and click Continue.

5. Choose Select plugins to install.

   You will need the following plugins:

   • Pipeline: Plugin that retrieves source code from the version control system to build, test, and deploy the code.
   • Git: Plugin for working with Git repositories.
   • Credentials Binding: Plugin for creating environment variables with authentication data.

6. Click Install. This will start the installation of the components you selected.

7. After the installation is complete, you will be prompted to create an administrator account. Fill in the form and click Save and Continue.

8. You will be prompted to create a URL for Jenkins. Leave this URL format: http://<VM_public_IP_address>/. Click Save and finish.

9. Click Start using Jenkins to complete the installation and go to the Jenkins administration panel.

Set up a Jenkins taskSet up a Jenkins task

Enter your Yandex Cloud credentials and create a task to download changes from the GitHub repository so that Jenkins can build images. These credentials will be used in the variables stored in the Packer configuration files.

1. Open the Jenkins administration panel.
2. In the top-right corner, click the username.
3. Select Credentials.
4. Under Stores scoped to Jenkins, click the Global link.
5. Get the ID of the subnet where the images will be built by running the yc vpc subnet list command.
6. Click Add credentials. Specify the following parameters:
   1. From the Kind list, select Secret text.
   2. In the Scope list, leave Global.
   3. In the Secret field, enter the ID of your folder.
   4. In the Id field, specify YC_FOLDER_ID. Click OK.
7. Create another secret with these parameters:
   1. Kind: Secret text.
   2. Scope: Global.
   3. Secret: ID of the subnet where the Jenkins VM is hosted.
   4. ID: YC_SUBNET_ID.
8. Create another secret with these parameters:
   1. Kind: Secret file.
   2. Scope: Global.
   3. File: File named <username>.json from Step 1.
   4. ID: YC_ACCOUNT_KEY_FILE.
9. Go back to the main page of the administration panel and select New item.
10. Enter a name for the task: jenkins-tutorial; select Pipeline as the task type. Click OK.
11. In the window that opens, check the GitHub hook trigger for GITScm polling box. This option allows you to run the build every time you pushsomething to the master branch of your Git repository.
12. Under Pipeline, select Pipeline script from SCM from the Definition list.
13. From the SCM list, select Git.
14. In the Report URL field, enter the URL of your GitHub fork.
15. In the Script path field, enter Jenkinsfile.
16. Leave other fields unchanged and click Save.

Configure the GitHub repositoryConfigure the GitHub repository

In the GitHub repository settings, enable a webhook to initiate a Jenkins build and add the public SSH key for authorization.

Enable a webhookEnable a webhook

1. In your browser, open the fork of your GitHub repository.
2. Click the Settings tab.
3. Select Webhooks and click Add webhook.
4. In the Payload URL field, enter http://<VM_public_IP_address>/github-webhook/.
5. Click Add webhook.

Add an SSH key to GitHubAdd an SSH key to GitHub

1. Click your GitHub avatar. In the menu that opens, select Settings.
2. Click SSH and GPG keys.
3. Click New SSH key.
4. In the Title field, enter a name for your key.
5. Paste your SSH key into the Key box.
6. Click Add SSH key.

Create an image using JenkinsCreate an image using Jenkins

Jenkins launches an image build automatically after you run push in the master branch of your GitHub repository.

1. Clone the examples repository fork you created when getting started to your computer:

   git clone https://github.com/<GitHub_login>/yc-marketplace-vm-image-packer.git
   

2. Modify the Packer templates in the packer directory. You can find articles and guides regarding Packer templates on the Packer website. In the image_family and source_image_family parameters, specify the image families for Jenkins to build.

3. Modify the Pipeline for Jenkinsfile located in the repository root directory. For the Pipeline user handbook, see the Packer website.

4. Upload the changes to GitHub:

   git add -A
   git commit -m "Build update"
   git push
   

5. Open the Jenkins administration panel and check the task status.

6. If the configuration is correct, the system will start building images. You can see the result in the build logs.

As a result, three new images will appear in Compute Cloud under Images:

• Debian: Basic image with the latest updates.
• Nginx: Debian-based image with an nginx web server.
• Django: Debian-based image with the Django framework.

Deploy the imagesDeploy the images

Once the images have been created, you can use them to create your VMs. Create a test infrastructure using Terraform:

1. In the fork directory, go to the Terraform files directory:

   cd examples/jenkins-packer/terraform
   

2. Rename the terraform.tfvars_example file:

   mv terraform.tfvars_example terraform.tfvars
   

3. Provide the required values in the file fields. See also the Terraform and Yandex Cloud overview articles.

4. Initialize the Terraform provider by running terraform init.

5. Run the terraform plan -var-file="terraform.tfvars" command. Check the configuration you created.

6. Run terraform apply and confirm that you want to create the infrastructure by typing yes into the terminal prompt.

This will create:

1. Cloud network.
2. Subnets in all availability zones.
3. VMs from the images created by Packer. VMs with nginx will get public IP addresses. All VMs will be connected to subnets.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

• Delete the VMs you created.
• Delete the images you created.
• Delete the service account and the <username.json> file.
• Delete the network and the subnets.

To delete the resources created with Terraform, run terraform destroy.