Locking Terraform states using Yandex Managed Service for YDB

Yandex Cloud supports infrastructure management through Terraform. To allow multiple users to manage the infrastructure, you can automatically upload the Terraform states and store them in Yandex Object Storage.

When multiple users try to access the same state from Object Storage at the same time, conflicts may occur. To prevent such conflicts, you can deploy a database in Yandex Managed Service for YDB and use it to implement Terraform's native state locking mechanism. Every time you use Terraform to update the infrastructure, the state will be automatically locked until the update is applied.

To set up storing Terraform states in Object Storage and locking them by Managed Service for YDB:

1. Prepare your cloud.
2. Create a service account and static access key.
3. Create a bucket.
4. Create a Managed Service for YDB database.
5. Install and configure Terraform.
6. Configure the backend.
7. Deploy the configuration.
8. Check the saved state.
9. Check whether the state is locked.

If you no longer need the resources you created, delete them.

Terraform and its providers are distributed under the Business Source License.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The infrastructure support cost for Terraform states includes:

• Data storage fees (see Object Storage pricing).
• Fee for running queries to the database (see Managed Service for YDB pricing).

The cost of support for the example infrastructure deployed through Terraform in this tutorial includes:

• Fee for a continuously running VM (see Yandex Compute Cloud pricing).
• Fee for using a dynamic public IP address (see Yandex Virtual Private Cloud pricing).

If you deploy resources of other Yandex Cloud services, the cost will change according to the relevant service plans.

Create a service account and static access keyCreate a service account and static access key

1. Create a service account with the storage.editor and ydb.admin roles for the folder specified in the provider settings.
2. Get a static access key. Save the key ID and secret key: you will need them at the next steps.

Create a bucketCreate a bucket

Create a bucket with restricted access. This bucket will store the Terraform state file.

Create a Managed Service for YDB databaseCreate a Managed Service for YDB database

Create a Serverless DB named state-lock-db:

Create a tableCreate a table

Install and configure TerraformInstall and configure Terraform

Install TerraformInstall Terraform

Get the authentication credentialsGet the authentication credentials

Use a service account to manage the Yandex Cloud infrastructure using Terraform. It will help you flexibly configure access permissions to resources.

You can also access Terraform from your Yandex account,, or a federated account, but this method is less secure. For more information, see the end of this section.

1. If you do not have the Yandex Cloud command line interface, install it.

2. Set up the CLI profile to run operations on behalf of the service account:

   CLI

   1. Create an authorized key for your service account and save the file:

      yc iam key create \
        --service-account-id <service_account_ID> \
        --folder-name <service_account_folder_name> \
        --output key.json
      

      Where:

      • service-account-id: Service account ID.
      • folder-name: Name of the folder in which the service account was created.
      • output: Name of the file with the authorized key.

      Result:

      id: aje8nn871qo4********
      service_account_id: ajehr0to1g8b********
      created_at: "2022-09-14T09:11:43.479156798Z"
      key_algorithm: RSA_2048
      

   2. Create a CLI profile to run operations on behalf of the service account. Name the profile:

      yc config profile create <profile_name>
      

      Result:

      Profile 'sa-terraform' created and activated
      

   3. Set the profile configuration:

      yc config set service-account-key key.json
      yc config set cloud-id <cloud_ID>
      yc config set folder-id <folder_ID>
      

      Where:

      • service-account-key: File with the service account authorized key.
      • cloud-id: Cloud ID.
      • folder-id: Folder ID.

3. Add the credentials to the environment variables:

   Bash

   PowerShell

   export YC_TOKEN=$(yc iam create-token)
   export YC_CLOUD_ID=$(yc config get cloud-id)
   export YC_FOLDER_ID=$(yc config get folder-id)
   

   Where:

   • YC_TOKEN: IAM token.
   • YC_CLOUD_ID: Cloud ID.
   • YC_FOLDER_ID: Folder ID.

   $Env:YC_TOKEN=$(yc iam create-token)
   $Env:YC_CLOUD_ID=$(yc config get cloud-id)
   $Env:YC_FOLDER_ID=$(yc config get folder-id)
   

   Where:

   • YC_TOKEN: IAM token.
   • YC_CLOUD_ID: Cloud ID.
   • YC_FOLDER_ID: Folder ID.

   Note

   The IAM token lifetime does not exceed 12 hours; however, we recommend requesting it more often, such as once per hour.

export YC_TOKEN=$(yc iam create-token)
export YC_CLOUD_ID=$(yc config get cloud-id)
export YC_FOLDER_ID=$(yc config get folder-id)

$Env:YC_TOKEN=$(yc iam create-token)
$Env:YC_CLOUD_ID=$(yc config get cloud-id)
$Env:YC_FOLDER_ID=$(yc config get folder-id)

Create a Terraform configuration fileCreate a Terraform configuration file

1. Create a directory with any name, for example, cloud-terraform. It will store the Terraform configuration files.
2. Create a configuration file with the .tf extension in this directory, for example, example.tf.

Configure a providerConfigure a provider

1. If you previously configured a provider from the HashiCorp registry, save its settings:

   Linux/macOS

   Windows

   mv ~/.terraformrc ~/.terraformrc.old
   

   mv $env:APPDATA/terraform.rc $env:APPDATA/terraform.rc.old
   

2. Specify the source the provider will be installed from.

   Linux/macOS

   Windows

   Open the Terraform CLI configuration file:

   nano ~/.terraformrc
   

   Note

   The .terraformrc file must be in the user's home root folder, e.g., /home/user/ or /User/user/.

   Open the Terraform CLI configuration file named terraform.rc in your user's %APPDATA% folder.

   To find out the absolute path to the %APPDATA% folder, run the echo %APPDATA% command for cmd or the $env:APPDATA command for PowerShell.

   Add the following section to the file:

   provider_installation {
     network_mirror {
       url = "https://terraform-mirror.yandexcloud.net/"
       include = ["registry.terraform.io/*/*"]
     }
     direct {
       exclude = ["registry.terraform.io/*/*"]
     }
   }
   

   For more information about setting up mirrors, see the documentation.

3. Add the following sections at the top of the .tf configuration file:

   terraform {
     required_providers {
       yandex = {
         source = "yandex-cloud/yandex"
       }
     }
     required_version = ">= 0.13"
   }
   
   provider "yandex" {
     zone = "<default_availability_zone>"
   }
   

   Where:

   • source: Provider's global source address.
   • required_version: Minimum Terraform version the provider is compatible with.
   • provider: Provider name.
   • zone: Availability zone where all cloud resources will be created by default.

4. Run the terraform init command in the folder containing the .tf configuration file. This command initializes the providers specified in the configuration files and allows you to work with the provider resources and data sources.

If the provider installation failed, create a support request stating provider name and version.

If you used the .terraform.lock.hcl file, prior to the initialization, run the terraform providers lock command specifying the URL of the mirror to upload the provider from and the platforms the configuration will run on:

terraform providers lock -net-mirror=https://terraform-mirror.yandexcloud.net -platform=<platform_1_name> -platform=<platform_2_name> yandex-cloud/yandex

Where:

• -net-mirror: Address of the mirror to upload the provider from.
• -platform: Platforms to use the configuration on. The possible values include:
  • windows_amd64: 64-bit Windows
  • linux_amd64: 64-bit Linux
  • darwin_arm64: 64-bit macOS

If you used the Terraform modules, first run terraform init, then delete the lock file. After that, run the terraform providers lock command.

For more information about the terraform providers lock command, see the Terraform documentation.

Configure the backendConfigure the backend

To save the Terraform state in Object Storage and activate state locking:

1. Add the previously obtained key ID and secret key to environment variables:

   Bash

   PowerShell

   export ACCESS_KEY="<key_ID>"
   export SECRET_KEY="<secret_key>"
   

   $Env:AWS_ACCESS_KEY_ID="<key_ID>"
   $Env:AWS_SECRET_ACCESS_KEY="<secret_key>"
   

2. Add provider and backend settings to the configuration file:

   terraform {
     required_providers {
       yandex = {
         source = "yandex-cloud/yandex"
       }
     }
     required_version = ">= 0.13"
   
     backend "s3" {
       endpoints = {
         s3       = "https://storage.yandexcloud.net"
         dynamodb = "<Document_API_DB_endpoint>"
       }
       bucket            = "<bucket_name>"
       region            = "ru-central1"
       key               = "<path_to_state_file_in_bucket>/<state_file_name>.tfstate"
   
       dynamodb_table = "<table_name>"
   
       skip_region_validation      = true
       skip_credentials_validation = true
       skip_requesting_account_id  = true # This option is required for Terraform 1.6.1 or higher.
       skip_s3_checksum            = true # This option is required to describe backend for Terraform version 1.6.3 or higher.
     }
   }
   
   provider "yandex" {
     zone = "<default_availability_zone>"
   }
   

   Where:

   • bucket: Bucket name.
   • dynamodb: Document API DB in https://docapi.serverless.yandexcloud.net/ru-central1/b1gia87mbaom******** format.
   • key: Object key in the bucket (name and path to the Terraform state file in the bucket).
   • dynamodb_table: Table name.

   To read more about the state storage backend, see the Terraform website.

3. Run the following command in the folder with the configuration file:

   terraform init
   

Deploy the configurationDeploy the configuration

In this example, you will create a VM named terraform-vm that will be connected to the subnet-1 subnet in the ru-central1-a availability zone. This subnet will be in the network-1 cloud network.

The VM will have 2 cores and 4 GB RAM. It will be automatically assigned a public and a private IP address from the 192.168.10.0/24 range in subnet-1. The VM will run Ubuntu and host the public part of the key to enable SSH access.

1. Save the following configuration as a separate example-vm.tf file in the folder with the backend configuration file:

   resource "yandex_compute_image" "ubuntu_2004" {
     source_family = "ubuntu-2004-lts"
   }
   
   resource "yandex_compute_disk" "boot-disk" {
     name     = "boot-disk"
     type     = "network-hdd"
     zone     = "ru-central1-a"
     size     = "20"
     image_id = yandex_compute_image.ubuntu_2004.id
   }
   
   resource "yandex_compute_instance" "vm-1" {
     name = "terraform-vm"
   
     resources {
       cores  = 2
       memory = 4
     }
   
     boot_disk {
       disk_id = yandex_compute_disk.boot-disk.id
     }
   
     network_interface {
       subnet_id = yandex_vpc_subnet.subnet-1.id
       nat       = true
     }
   
     metadata = {
       user-data = "#cloud-config\nusers:\n  - name: <username>\n    groups: sudo\n    shell: /bin/bash\n    sudo: 'ALL=(ALL) NOPASSWD:ALL'\n    ssh-authorized-keys:\n      - ${file("<path_to_public_SSH_key>")}"
     }
   }
   
   resource "yandex_vpc_network" "network-1" {
     name = "network1"
   }
   
   resource "yandex_vpc_subnet" "subnet-1" {
     name           = "subnet1"
     zone           = "ru-central1-a"
     network_id     = yandex_vpc_network.network-1.id
     v4_cidr_blocks = ["192.168.10.0/24"]
   }
   
   output "internal_ip_address_vm_1" {
     value = yandex_compute_instance.vm-1.network_interface.0.ip_address
   }
   
   output "external_ip_address_vm_1" {
     value = yandex_compute_instance.vm-1.network_interface.0.nat_ip_address
   }
   
   output "subnet-1" {
     value = yandex_vpc_subnet.subnet-1.id
   }
   

2. Check the configuration using the terraform plan command.

3. Deploy the configuration using the terraform apply command.

Check the saved stateCheck the saved state

Make sure the state file is uploaded to Yandex Object Storage:

Check the state lockCheck the state lock

Try to update the infrastructure concurrently with another user. If the lock mechanism works, Terraform will return the following message after running terraform apply:

member Error: Error acquiring the state lock

member Error message: ConditionalCheckFailedException: Condition not satisfied
member Lock Info:
member   ID:        <...>
member   Path:      terraform-object-storage-tutorial/TF/cloud-storage.tfstate
member   Operation: OperationTypeApply
member   Who:       LD\user@i7293
member   Version:   1.4.2
member   Created:   <...>
member   Info:

member Terraform acquires a state lock to protect the state from being written
member by multiple users at the same time. Please resolve the issue above and try
member again. For most commands, you can disable locking with the "-lock=false"
member flag, but this is not recommended.

How to delete the resources you createdHow to delete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the table from the database.
2. Delete the state-lock-db database.
3. Delete the bucket.

See alsoSee also

• Getting started with Terraform.
• Uploading Terraform states to Object Storage.
• Using Yandex Cloud modules in Terraform.
• Terraform data sources.