Connecting to Object Storage from Virtual Private Cloud

In Yandex Cloud, you can connect to Yandex Object Storage via the appropriate API endpoint whose FQDN is then translated to a public IP by means of DNS.

This article describes how to deploy a cloud infrastructure in Yandex Cloud to set up access to Object Storage for resources that are hosted in a VPC cloud network and have no public IP addresses or internet access through a NAT gateway.

After the solution is deployed in Yandex Cloud, the following resources will be created:

For a cloud network with the resources hosted in Cloud DNS, create thestorage.yandexcloud.net. internal DNS zone and a type A resource record that maps the storage.yandexcloud.net domain name of Object Storage to the IP address of the internal network load balancer. With this record, traffic from the cloud resources to Object Storage will be routed to the internal load balancer that will distribute the load across the NAT instances.

To deploy the NAT instances, use the NAT instance based on Ubuntu 22.04 LTS image from Cloud Marketplace. It translates source and target IP addresses to ensure traffic routing to the Object Storage public IP address.

By placing the NAT instances in multiple availability zones, you can ensure fault-tolerant access to Object Storage. By increasing the number of NAT instances, you can scale the solution up if the workload increases. When calculating the number of NAT instances, factor in the locality of traffic handling by the internal load balancer.

Object Storage access policies allow actions involving buckets only from the public IP addresses of NAT instances. Only cloud resources that use this solution can access the bucket. You cannot connect to a bucket in Object Storage via a public API endpoint. You can disable this limitation in the Terraform configuration file, if required.

Test results for NAT instance throughputTest results for NAT instance throughput

The solution was tested on a single NAT instance with the following configuration:

• Platform: Intel Ice Lake (standard-v3)
• Performance level: 100%
• vCPU cores: 2
• RAM: 2 GB

The average Object Storage traffic processing speed was 250 MB/s, both egress and ingress.

The test was performed using the warp tool on VM instances that could access Object Storage through this solution. The warp command used for the test had the following parameters:

warp get \
  --host storage.yandexcloud.net \
  --access-key <static_key_ID> \
  --secret-key <secret_key> \
  --tls \
  --bucket <bucket_name> \
  --obj.randsize \
  --concurrent 20 \
  --warp-client <warp_client_IP_addresses>

Tips for solution deployment in the production environmentTips for solution deployment in the production environment

• When deploying your NAT instances in multiple availability zones, set an even number of VMs to evenly distribute them across the availability zones.

• When selecting the number of NAT instances, factor in the locality of traffic handling by the internal load balancer.

• Once the solution is deployed, only reduce the number of NAT instances or update the list of availability zones in the yc_availability_zones parameter during a pre-scheduled time window. When the changes are being applied, traffic handling may be interrupted.

• If a NAT instance demonstrates a high CPU steal time value as the Object Storage workload increases, we recommend enabling a software-accelerated network for that NAT instance.

• By default, buckets in Object Storage can be accessed via the Yandex Cloud management console. You can revoke this permission using the bucket_console_access = false parameter.

• If you skip mgmt_ip with bucket_private_access = true, solution deployment using Terraform on a workstation will fail with a bucket access error.

• If you are using your own DNS server, create type A resource records in its settings in the following format:

  Name	Type	Value
  storage.yandexcloud.net	A	<internal_load_balancer_IP_address>
  <bucket_name>.storage.yandexcloud.net	A	<internal_load_balancer_IP_address>

• Save the pt_key.pem private SSH key for connecting to the NAT instances to a secure location or recreate it separately from Terraform.

• Once the solution is deployed, SSH access to the NAT instances will be disabled. To enable access to the NAT instances over SSH, add a rule for inbound SSH traffic (TCP/22) in the s3-nat-sg security group to enable access only from certain IP addresses of admin workstations.

Deployment planDeployment plan

To deploy the solution and test it:

1. Get your cloud ready.
2. Set up an environment for deploying the resources.
3. Deploy the solution.
4. Test the solution.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The infrastructure support costs include:

• Fee for using Object Storage (see Yandex Object Storage pricing).
• Fee for using a network load balancer (see Network Load Balancer pricing).
• Fee for continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for using public IP addresses and outbound traffic (see Yandex Virtual Private Cloud pricing).

Set up an environment for deploying the resourcesSet up an environment for deploying the resources

1. Install Terraform.

2. If you do not have the Yandex Cloud command line interface yet, install it and sign in.

3. Check if there is an account in the Yandex Cloud-enabled cloud with admin permissions for the folder where you are deploying the solution.

4. Install Git.

5. Check whether your cloud quotas allow you to deploy your resources for this scenario:

   Information about the number of new resources

   Resource	Amount
   Virtual machines	5
   VM vCPUs	10
   VM RAM	10 GB
   Disks	5
   HDD size	30 GB
   SSD size	40 GB
   Network load balancer	1
   Target group for the load balancer	1
   Networks	11
   Subnets	3
   Static public IP addresses	4
   Security groups	1
   DNS zone	1
   Bucket	1
   Service account	2
   Static key for the service account	1

   ¹ If the user did not specify the ID of an existing network in terraform.tfvars.

Deploy the solution using TerraformDeploy the solution using Terraform

1. Clone the yandex-cloud-examples/yc-s3-private-endpoint repository to your workstation and navigate to the yc-s3-private-endpoint folder:

   git clone https://github.com/yandex-cloud-examples/yc-s3-private-endpoint.git
   cd yc-s3-private-endpoint
   

2. Set up an environment for authentication in Terraform (to learn more, see Getting started with Terraform):

   export YC_TOKEN=$(yc iam create-token)
   

3. The variables.tf file defines the variable parameters of the resources to create. Insert your custom variable values into the terraform.tfvars file. Refer to the table below to see which parameters need changing.

   Detailed information about the values to set

   Name of parameter	Replace with a custom value	Description	Type	Example
   folder_id	Yes	ID of the folder to host the solution components.	string	"b1gentmqf1ve********"
   vpc_id	—	ID of the cloud network for which access to Object Storage is set up. If not specified, a new network will be created.	string	"enp48c1ndilt********"
   yc_availability_zones	—	List of the availability zones for deploying NAT instances.	list(string)	["ru-central1-a", "ru-central1-b"]
   subnet_prefix_list	—	List of prefixes of cloud subnets to host the NAT instances (one subnet per each availability zone from the yc_availability_zones list in the following order: ru-central1-a, ru-central1-b, and so on).	list(string)	["10.10.1.0/24", "10.10.2.0/24"]
   nat_instances_count	—	Number of NAT instances to deploy. We recommend setting an even number to evenly distribute the instances across the availability zones.	number	4
   bucket_private_access	—	Only allow bucket access from the public IP addresses of NAT instances. If true, access is limited. To remove the limit, set false.	bool	true
   bucket_console_access	—	Allow bucket access via the Yandex Cloud management console. If true, access is allowed. To deny access, set false. This parameter is required if bucket_private_access is set to true.	bool	true
   mgmt_ip	Yes	Public IP address of your workstation where you are deploying the infrastructure using Terraform. It is used to allow your workstation to perform actions involving the bucket when deploying Terraform. This parameter is required if bucket_private_access is set to true.	string	"A.A.A.A"
   trusted_cloud_nets	Yes	List of aggregated prefixes of cloud subnets for which Object Storage access is allowed. It is used in the inbound traffic rule of the security groups for the NAT instances.	list(string)	["10.0.0.0/8", "192.168.0.0/16"]
   vm_username	—	NAT instance and test VM user names	string	"admin"
   s3_ip	No	Object Storage public IP address	string	213.180.193.243
   s3_fqdn	No	Object Storage domain name	string	storage.yandexcloud.net

4. Initialize Terraform:

   terraform init
   

5. Check the list of cloud resources you are about to create:

   terraform plan
   

6. Create resources:

   terraform apply
   

7. Once the terraform apply process is completed, the command line will output information required for connecting to the test VM and running test operations with Object Storage. Later on, you can view this information by running the terraform output command:

   Information about the deployed resources

   Name	Description	Sample value
   path_for_private_ssh_key	File with a private key used to connect to the NAT instances and test VM over SSH	./pt_key.pem
   vm_username	NAT instance and test VM user names	admin
   test_vm_password	admin password for the test VM	v3RCqU****
   s3_bucket_name	Bucket name in Object Storage	s3-bucket-<...>
   s3_nlb_ip_address	IP address of the internal load balancer	10.10.1.100

Test the solutionTest the solution

1. In the management console, navigate to the folder where you created the resources.

2. Select Compute Cloud.

3. Select test-s3-vm from the list of VM instances.

4. Navigate to the Serial console tab.

5. Click Connect.

6. Enter the admin username and the password from the terraform output test_vm_password command output (without quotation marks).

7. Run this command:

   dig storage.yandexcloud.net
   

8. Make sure the Object Storage domain name in the DNS server response matches the IP address of the internal load balancer. The output of the type A resource record is as follows:

   ;; ANSWER SECTION:
   storage.yandexcloud.net. 300    IN      A       10.10.1.100
   

9. Get an object from the bucket in Object Storage using the AWS CLI. The bucket name will be fetched from the test VM environment variable.

   aws --endpoint-url=https://storage.yandexcloud.net \
   s3 cp s3://$BUCKET/s3_test_file.txt s3_test_file.txt
   

   Result:

   download: s3://<bucket_name>/s3_test_file.txt to ./s3_test_file.txt
   

10. You can additionally run a number of commands to test Object Storage. The bucket name will be fetched from the test VM environment variable.

    Upload the downloaded test file to the bucket under a different name:

    aws --endpoint-url=https://storage.yandexcloud.net \
    s3 cp s3_test_file.txt s3://$BUCKET/textfile.txt
    

    Result:

    upload: ./s3_test_file.txt to s3://<bucket_name>/textfile.txt
    

    Get a list of objects in the bucket:

    aws --endpoint-url=https://storage.yandexcloud.net \
    s3 ls --recursive s3://$BUCKET
    

    Result:

    2023-08-16 18:24:05         53 s3_test_file.txt \
    2023-08-16 18:41:39         53 textfile.txt
    

    Delete the object you uploaded to the bucket:

    aws --endpoint-url=https://storage.yandexcloud.net \
    s3 rm s3://$BUCKET/textfile.txt
    

    Result:

    delete: s3://<bucket_name>/textfile.txt
    

Delete the resources you createdDelete the resources you created

To delete the resources created with Terraform:

1. In the terminal window, go to the directory containing the infrastructure plan.

   Warning

   Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

2. Delete resources:

   1. Run this command:

      terraform destroy
      

   2. Confirm deleting the resources and wait for the operation to complete.

   All the resources described in the Terraform manifests will be deleted.