Automating image builds using Jenkins and Packer
Based on the specified configuration, Packer
You can use images to create a cloud infrastructure, for example, using Terraform
To install and configure Jenkins, Packer, GitHub, and Terraform to work together:
- Prepare your cloud.
- Configure the environment.
- Create a service account.
- Create a VM with Jenkins.
- Install Packer to your VM.
- Configure Jenkins.
- Set up a Jenkins task.
- Configure the GitHub repository.
- Create an image using Jenkins.
- Deploy the images using Terraform.
If you no longer need the VM or the cluster you created, delete them.
Prepare your cloud
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
The infrastructure support costs include:
- Fee for continuously running VMs (see Compute Cloud pricing).
- Fee for storing created images (see Compute Cloud pricing).
- Fee for using a dynamic public IP address (see Yandex Virtual Private Cloud pricing).
Configure the environment
Set up the software:
- Install the Yandex Cloud command line interface.
- Install
Terraform. See also Getting started with Terraform. - Download
the jq utility. - Configure
Git. If you are running Windows, use Git Bash. - Create
a repository branch with examples in your GitHub account. - Prepare an SSH key for VM access.
Create a service account
Jenkins uses service accounts to perform actions in your cloud and folder. To create a service account:
-
Get the IDs of the folder and cloud by running
yc config list
. -
Create a service account and provide its ID to the environment variable using the following commands:
yc iam service-account create --name <username> yc iam key create --service-account-name <username> -o <username.json> SERVICE_ACCOUNT_ID=$(yc iam service-account get --name <username> --format json | jq -r .id)
This will create a JSON file containing login credentials in the current folder.
-
Assign the service account the
admin
role for the folder the operations will be performed in:yc resource-manager folder add-access-binding <folder_name> --role admin --subject serviceAccount:$SERVICE_ACCOUNT_ID
Create a VM with Jenkins
Jenkins will get VM image configuration changes from GitHub and then use Packer to create images in the cloud.
To create a VM with Jenkins:
-
In the management console
, select the folder to create your VM in. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Under Boot disk image, go to the Marketplace tab, click Show all Marketplace products, and select the Jenkins image.
Note
If you are configuring the Jenkins VM manually, follow the procedure
. -
Under Location, select an availability zone to place your VM in.
-
Under Disks and file storages, enter
15 GB
as your boot disk size. -
Under Computing resources, navigate to the Custom tab and specify the parameters as follows:
- Platform:
Intel Ice Lake
- vCPU:
2
- Guaranteed vCPU performance:
20%
- RAM:
2 GB
- Platform:
-
Under Network settings:
-
In the Subnet field, enter the ID of a subnet in the new VM’s availability zone. Alternatively, you can select a cloud network from the list.
-
Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.
-
If you do not have a network, click Create network to create one:
- In the window that opens, enter the network name and select the folder to host the network.
- (Optional) Select the Create subnets option to automatically create subnets in all availability zones.
- Click Create network.
-
-
In the Public IP field, select
Auto
to assign the VM a random external IP address from the Yandex Cloud pool or select a static address from the list if you reserved one in advance.
-
-
Under Access, specify the information required to access the VM:
-
In the Login field, enter the name of the user you want to create on the VM, e.g.,
yc-user
.Alert
Do not use
root
or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use thesudo
command. -
In the SSH key field, paste the contents of the public key file.
You need to create a key pair for the SSH connection yourself. To learn how, see Connecting to a VM via SSH.
-
-
Under General information, specify the VM name:
jenkins-tutorial
. -
Click Create VM.
Install Packer
Packer enables you to create VM disk images with parameters specified in a configuration file.
Note
Yandex Cloud requires Packer 1.5 or higher.
-
Download a Packer
distribution for Linux. You can also download a Packer distribution for your platform from a mirror . -
Upload Packer to the VM you created:
scp packer_<Packer_version>_linux_amd64.zip <login>@<VM_public_IP_address>:~/
-
Connect to the VM via SSH. To do this, use
ssh
in Linux or macOS, orPuTTY
in Windows. -
Create a new folder, move the Packer executables there, and unpack the archive:
sudo mkdir /opt/yandex-packer/ sudo mv packer_<Packer_version>_linux_amd64.zip /opt/yandex-packer/ unzip packer_<Packer_version>_linux_amd64.zip
-
Configure the Yandex Compute Builder plugin
:-
Create a file named
config.pkr.hcl
in the/opt/yandex-packer/
directory with the following contents:packer { required_plugins { yandex = { version = ">= 1.1.2" source = "github.com/hashicorp/yandex" } } }
-
Install the plugin:
packer init <path_to_config.pkr.hcl>
Result:
Installed plugin github.com/hashicorp/yandex v1.1.2 in ...
-
-
All Jenkins system activities will be performed on behalf of the
jenkins
user. Grant this user rights to launch Packer:sudo chmod u+x /opt/yandex-packer/packer* sudo chown jenkins:jenkins /opt/yandex-packer/packer*
Configure Jenkins
To build images based on configurations from GitHub, configure Jenkins:
-
Connect to the VM via SSH. To do this, use
ssh
in Linux or macOS, orPuTTY
in Windows. -
Open the password file needed to launch the configuration and copy the password:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
-
In your browser, go to
https://<public_IP_address_of_VM_with_Jenkins>
. Open the Jenkins management console. -
Enter the copied password in the Administrator password field and click Continue.
-
Choose Select plugins to install.
You will need the following plugins:
Pipeline
: Plugin that retrieves source code from the version control system to build, test, and deploy the code.Git
: Plugin for working with git repositories.Credentials Binding
: Plugin for creating environment variables containing authentication data.
-
Click Install. This will start the installation of the selected components.
-
When the installation is complete, you will be prompted to create an administrator account. Fill in the form and click Save and Continue.
-
You will be prompted to create a URL for Jenkins. Leave this URL format:
http://<VM_public_IP_address>/
. Click Save and finish. -
Click Start using Jenkins to complete the installation and go to the Jenkins administration panel.
Set up a Jenkins task
Enter your Yandex Cloud authorization data and create a task to download changes from the github repository so that Jenkins can build images. The authorization data is used in the variables stored in the Packer configuration files.
- Open the Jenkins administration panel.
- In the top-right corner, click the username.
- Select Credentials.
- Under Stores scoped to Jenkins, click the
Global
link. - Get the ID of the subnet the images will be built in by running the
yc vpc subnet list
command. - Click Add credentials. Specify the following parameters:
- In the Kind list, select
Secret text
. - In the Scope list, leave
Global
. - In the Secret field, enter the ID of your folder.
- In the Id field, specify
YC_FOLDER_ID
. Click OK.
- In the Kind list, select
- Create another secret with the following parameters:
- Kind:
Secret text
. - Scope:
Global
. - Secret: ID of the subnet where the Jenkins VM is hosted.
- ID:
YC_SUBNET_ID
.
- Kind:
- Create another secret with the following parameters:
- Kind:
Secret file
. - Scope:
Global
. - File: File named
<username>.json
from Step 1. - ID:
YC_ACCOUNT_KEY_FILE
.
- Kind:
- Go back to the main page of the administration panel and select New item.
- Enter a name for the task:
jenkins-tutorial
and select Pipeline as the task type. Click OK. - In the window that opens, select the GitHub hook trigger for GITScm polling checkbox. This option allows you to run the build every time you
push
to themaster
branch of your git repository. - Under Pipeline, select
Pipeline script from SCM
from the Definition list. - In the SCM list, select
Git
. - In the Report URL field, enter the URL of your GitHub fork.
- In the Script path field, enter
jenkins-packer/Jenkinsfile
. - Leave the other fields unchanged and click Save.
Configure GitHub repository
In the GitHub repository settings, enable a webhook to initiate a Jenkins build and add the public SSH key for authorization.
Enable Webhook
- Open the fork of your GitHub repository in the browser.
- Click the Settings tab.
- Select Webhooks and click Add webhook.
- In the Payload URL field, enter
http://<VM_public_IP_address>/github-webhook/
. - Click Add webhook.
Add an SSH key to GitHub
- Click your GitHub avatar. In the menu that opens, select Settings.
- Click SSH and GPG keys.
- Click New SSH key.
- In the Title field, enter a name for your key.
- Paste your SSH key into the Key box.
- Click Add SSH key.
Create an image using Jenkins
Jenkins launches an image build automatically after you run push
in the master
branch of your GitHub repository.
-
Clone the examples
repository fork you created while getting started, to your computer:git clone https://github.com/<GitHub_login>/examples.git
-
Make changes to the Packer templates in the
jenkins-packer/packer/
folder. You can find Packer template documentation on the developer's website . In theimage_family
andsource_image_family
parameters, specify the families of images to be built by Jenkins. -
Make changes to the Pipeline for Jenkins description file
Jenkinsfile
located in the repository root folder. For the Pipeline documentation, see the developer's website . -
Upload the changes to Github:
git add -A git commit -m "Build update" git push
-
Open the Jenkins administration panel and check the task status.
-
If all the settings are correct, image building will start. You can see the result in the build logs.
Note
When configuring a Jenkins task under GitHub Hook log, you may encounter a Polling has not run yet
error. If this happens, you should run the first build manually.
As a result, three new images will appear in Compute Cloud under Images:
Debian
: Basic image with the latest updates.Nginx
:Debian
-based image with an nginx web server.Django
:Debian
-based image with the Django framework.
Deploy the images
Once the images are created, you can use them to create your VMs. Create a test infrastructure using Terraform:
-
In the folder with the fork, go to the directory with the Terraform files:
cd examples/jenkins-packer/terraform
-
Rename the
terraform.tfvars_example
file:mv terraform.tfvars_example terraform.tfvars
-
Provide the required values in the file's fields. See also the Terraform
and Yandex Cloud provider documentation. -
Initialize the Terraform provider by running
terraform init
. -
Run the
terraform plan -var-file="terraform.tfvars"
command. Check the created configuration. -
Run
terraform apply
and confirm that you want to create the infrastructure by typingyes
into the terminal prompt.
This will create:
- Cloud network.
- Subnets in all availability zones.
- VMs from images created by Packer. VMs with nginx will get public IP addresses. All VMs will be connected to subnets.
Delete the resources you created
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:
- Delete the created VMs.
- Delete the created images.
- Delete the service account and the file named
<username.json>
. - Delete the network and the subnets.
To delete the resources created with Terraform, run terraform destroy
.