Configuring a fault-tolerant architecture in Yandex Cloud

In this tutorial, you will learn how to configure a fault-tolerant architecture in Yandex Cloud and check how it works using various test cases.

Fault tolerance is the property that enables a system to continue operating after a failure of one or more of its components.

To configure and test the architecture:

1. Prepare your cloud.
2. Set up a test bench.
3. Run test scenarios.

If you no longer need the resources you created, delete them.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost for supporting a fault-tolerant Yandex Cloud architecture includes:

• Fee for disks and continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for a continuously running Yandex Managed Service for PostgreSQL cluster (see Managed Service for PostgreSQL pricing).
• Fee for using a dynamic or static public IP address (see Yandex Virtual Private Cloud pricing).

Set up a test benchSet up a test bench

Description of the test bench:

• The application is packaged into a Docker image and pushed to Yandex Container Registry.

  Docker images are deployed on four VMs based on a Container Optimized Image. The VMs are grouped and located in two different availability zones.

• A DB cluster is managed by Managed Service for PostgreSQL and consists of two hosts that reside in different availability zones.

• The Load Testing Tool application generates the Yandex Cloud Marketplace workload that is applied to Yandex Network Load Balancer. The network load balancer distributes traffic across VMs.

Create TodoList app containersCreate TodoList app containers

To prepare the application to run in Yandex Cloud:

1. Download and unpack the repository with the demo application source code, the Terraform specifications, and a script to simulate the application failure.

2. Go to the repository:

   cd yandex-cloud-fault-tolerance-demo-master/app
   

3. Get authenticated in Container Registry:

   yc container registry configure-docker
   

4. Create a registry:

   yc container registry create --name todo-registry
   

5. Create a Docker image with the v1 tag:

   docker build . --tag cr.yandex/<registry_ID>/todo-demo:v1 --platform linux/amd64
   

6. Create a Docker image with the v2 tag (to test the application update case):

   docker build . --build-arg COLOR_SCHEME=dark --tag cr.yandex/<registry_ID>/todo-demo:v2 --platform linux/amd64
   

7. Push the Docker images to Container Registry:

   docker push cr.yandex/<registry_ID>/todo-demo:v1
   docker push cr.yandex/<registry_ID>/todo-demo:v2
   

Deploy the infrastructureDeploy the infrastructure

To prepare the environment for running the application in Yandex Cloud:

1. Install Terraform.

2. Go to the directory with the environment specification:

   cd ../terraform/app
   

3. Initialize Terraform in the spec directory:

   terraform init
   

4. Save the folder ID to the YC_FOLDER variable and the IAM token to the YC_TOKEN variable:

   export YC_FOLDER=<folder_ID>
   export YC_TOKEN=$(yc iam create-token)
   

5. Generate a key to connect to a VM via SSH:

   ssh-keygen -t ed25519
   

6. In the file named app/todo-service.tf, specify the path to the public SSH key (the default value is ~/.ssh/id_ed25519.pub).

7. Check the cloud quotas to be able to deploy the resources you need.

   Information about resources to be created

   The following resources will be created:

   • Virtual Private Cloud network with three subnets in all availability zones.
   • Two service accounts:
     • Service account for managing a VM group with the editor role.
     • Service account for downloading a Docker image to a VM with the container-registry.images.puller role.
   • VM group of four Container Optimized Image-based VMs in the ru-central1-b and ru-central1-d availability zones.
   • Managed Service for PostgreSQL cluster with two hosts in the ru-central1-b and ru-central1-d availability zones.
   • Network load balancer to distribute traffic between the group's VMs.

8. Deploy and run the application:

   terraform apply -var yc_folder=$YC_FOLDER -var yc_token=$YC_TOKEN -var user=$USER
   

   Where:

   • yc_folder: Folder where the application will be deployed.
   • yc_token: IAM token of the user to deploy the application.

To access the application, go to the lb_address received after running the terraform apply command.

Create and run the Load Testing Tool appCreate and run the Load Testing Tool app

1. Go to the directory with the Load Testing Tool specification:

   cd ../tank
   

2. Initialize Terraform in the directory with the Load Testing Tool specification:

   terraform init
   

3. In the file named tank/main.tf, specify the path to the public and private SSH keys (the default values are ~/.ssh/id_ed25519.pub and ~/.ssh/id_ed25519).

4. Deploy and run the VM:

   terraform apply -var yc_folder=$YC_FOLDER -var yc_token=$YC_TOKEN -var user=$USER -var overload_token=<overload_token>
   

   Where:

   • yc_folder: Folder where Load Testing Tool will be deployed.
   • yc_token: IAM token of the user to deploy Load Testing Tool.
   • overload_token: Token to connect to <overload.yandex.net>. To get the token, log in, click your profile at the top right, and select My api token from the drop-down menu.

5. Connect to the created VM via SSH. The connection address is specified in the terraform apply command output:

   ssh <username>@<VM_IP_address>
   

6. Run Load Testing Tool:

   sudo yandex-tank -c load.yaml
   

7. Go to <overload.yandex.net> and find the running load there: Public tests → show my tests only.

Running scenariosRunning scenarios

VM failureVM failure

How the failure shows itself: The VM with the application is unavailable.

Possible causes:

• A failure of the physical host that the VM was running on.
• The VM with the application was deleted by mistake.

To simulate the failure, delete one of the instances in the group:

Test bench reaction:

1. The network load balancer and Instance Groups get information about the VM failure and exclude it from load balancing: traffic stops coming to this VM instance and is distributed across the remaining instances in the group.
2. Instance Groups is automatically restored.
   1. Deletes the unavailable VM instance (in this test case, it is already deleted and this step is skipped).
   2. Creates a new VM instance.
   3. Waits for the application to start on the VM instance.
   4. Adds the VM instance to load balancing.

The load balancer and Instance Groups require some time to detect the problem and disable traffic to the faulty VM instance. This may cause Connection Timeout errors (HTTP code 0 on the Quantities and HTTP codes charts in the Load Testing Tool monitoring app).

After disabling load balancing for the unavailable VM instance, the user load is handled correctly.

Application failureApplication failure

How the failure shows itself: The application doesn't respond in time or doesn't work correctly from the user's point of view.

Possible causes:

• A memory leak caused the application to fail.
• The application is unable continue due to DB connectivity loss.
• The application fails to handle requests due to heavy load.

According to health check settings, Instance Groups polls VM instances in the group over HTTP. When operating normally, accessing the /healthy endpoint returns the HTTP code 200. Otherwise, Instance Groups starts the recovery procedure.

To simulate the yandex-cloud-fault-tolerance-demo-master repository failure, run this script:

fail_random_host.sh <VM_group_ID>

A random VM from the group will start returning the HTTP code 503.

Test bench reaction:

1. Instance Groups receives information about the application failure and excludes the VM instance from load balancing: traffic stops coming to this instance and is distributed across the remaining instances in the group.
2. Instance Groups is automatically restored.
   1. Restarts the faulty VM instance.
   2. Waits for the application to start on the VM instance.
   3. Adds the VM instance to load balancing.

Instance Groups polls the instance several times before disabling traffic and starting the recovery procedure. This may cause Service Unavailable errors (HTTP code 503 on the Quantities and HTTP codes charts in the Load Testing Tool monitoring app).

After disabling load balancing for the faulty VM instance, the user load is handled correctly.

Availability zone failureAvailability zone failure

How the failure shows itself: Multiple VMs are unavailable in the same zone.

Possible causes:

• Data center downtime.
• Scheduled maintenance in the data center.

To move your resources to another data center:

Test bench reaction:

1. Instance Groups disables load balancing for the VMs in the ru-central1-b availability zone.
2. These VMs will be deleted while at the same time new VMs will be created in the ru-central1-d zone.
3. Instance Groups adds the created VMs to load balancing.

The number of VM instances that can be simultaneously created and deleted depends on the deployment policy.

When disabling load balancing for VMs, Connection Timeout errors may occur (HTTP code 0 on the Quantities and HTTP codes charts in the Load Testing Tool monitoring app).

After disabling load balancing for the VM instances, the user load is handled correctly.

Updating an applicationUpdating an application

To update the application:

Test bench reaction:

1. Instance Groups disables load balancing for two VMs with the outdated application version (the status of these VMs is RUNNING_OUTDATED).
2. Deletes these VMs while at the same time creating new VMs with the new app version.
3. Enables load balancing for the new VMs.
4. The actions are repeated for the remaining two VMs with the outdated app version.

Refresh the app page. If the network load balancer sends your request to a VM instance that's already updated, you'll see the app version with a dark color scheme.

The number of VM instances that can be simultaneously created and deleted depends on the deployment policy.

When disabling load balancing for VMs, Connection Timeout errors may occur (HTTP code 0 on the Quantities and HTTP codes charts in the Load Testing Tool monitoring app).

After disabling load balancing for the VM instances, the user load is handled correctly.

Scaling a DB configurationScaling a DB configuration

You may need to scale your DB if:

• Cluster host performance is insufficient to handle requests.
• The data requires more storage capacity.

To scale the DB configuration:

Managed Service for PostgreSQL will run the update command for the cluster.

When switching between the master and replica (at the beginning and end of the update process), an Internal Server Error may occur (HTTP code 500 on the Quantities and HTTP codes charts in the Load Testing Tool monitoring app).

After switching, the user load is handled correctly.

Deleting applications and environmentsDeleting applications and environments

To delete the Load Testing Tool app, go to the yandex-cloud-fault-tolerance-demo-master/terraform/tank folder and run the following command:

terraform destroy -var yc_folder=$YC_FOLDER -var yc_token=$YC_TOKEN -var user=$USER -var overload_token=not-used

To delete the TodoList app, go to the yandex-cloud-fault-tolerance-demo-master/terraform/app folder and run the following command:

terraform destroy -var yc_folder=$YC_FOLDER -var yc_token=$YC_TOKEN -var user=$USER