Configuring Yandex Cloud DNS for Yandex Managed Service for ClickHouse® cluster access from other cloud networks
Clients (e.g., virtual machines) in the same cloud network as the cluster can access the cluster's hosts using their fully qualified domain names (FQDNs). However, if the clients are in another cloud network, this will not be possible. This is because the required DNS service zones are created automatically and only support name resolution within a single cloud network.
Yandex Cloud DNS enables you to configure a DNS zone shared by multiple cloud networks. This makes it possible to access cluster hosts by their FQDNs even if the client and the cluster are on different cloud networks.
Warning
Cluster hosts with public access enabled will become inaccessible by their FQDNs from within the cloud networks with the shared DNS zone. However, such hosts will still be accessible by their FQDNs from the internet.
To set up a shared DNS zone:
- Create a zone in Cloud DNS.
- Make sure the DNS records migrate to the zone.
- Check cluster availability.
If you no longer need the resources you created, delete them.
The following is an example where:
- You place a cluster and a VM named
cluster-vm
in a cloud network namedcluster-net
. - You create a new cloud network named
vm-net
with a VM namedother-vm
. - You check the cluster for accessibility from these VMs.
- You create a DNS zone named
vpc-peering-zone
and perform a migration. - You check the cluster for accessibility from these VMs once again.
Getting started
Assign roles
Make sure you have access to cloud networks in the folders where the cluster-net
and vm-net
networks will reside.
You must have one of the following roles:
admin
editor
vpc.admin
For information on assigning roles, see the Yandex Identity and Access Management documentation.
Create a cluster
- Using the command line, navigate to the folder that will contain the Terraform configuration files with an infrastructure plan for the example. Create the directory if it does not exist.
- Create a multi-host Managed Service for ClickHouse® cluster using Terraform by following the example.
Create other resources
-
Describe the new
vm-net
cloud network with thevm-subnet-ru-central1-a
subnet: CIDR172.17.1.0/24
,ru-central1-a
availability zone. To do this, add the following lines to thenetworks.tf
configuration file:networks.tf
resource "yandex_vpc_network" "vm-net" { name = "vm-net" } resource "yandex_vpc_subnet" "vm-subnet-a" { name = "vm-subnet-ru-central1-a" zone = "ru-central1-a" network_id = yandex_vpc_network.vm-net.id v4_cidr_blocks = ["172.17.1.0/24"] }
-
Describe the security groups:
- Additional security group named
cluster-vm-sg
in thecluster-net
network for thecluster-vm
virtual machine. - Default security group named
other-vm-sg
in thevm-net
network for theother-vm
virtual machine.
To do this, add the following lines to the
security-groups.tf
configuration file:security-groups.tf
resource "yandex_vpc_security_group" "cluster-vm-sg" { name = "cluster-vm-sg" network_id = yandex_vpc_network.cluster-net.id ingress { description = "Allow any ingress traffic" protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } egress { description = "Allow any egress traffic" protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } } resource "yandex_vpc_default_security_group" "other-vm-sg" { network_id = yandex_vpc_network.vm-net.id ingress { description = "Allow any ingress traffic" protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } egress { description = "Allow any egress traffic" protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } }
Warning
The above security groups allow any traffic in any direction and serve for demonstration purposes. This enables you to install the utilities required for connection on the VM and to connect to the cluster.
For real-world tasks, configure the security groups to allow only essential traffic.
- Additional security group named
-
Describe the VMs to use for migration and DNS functionality check:
cluster-vm
: On thecluster-net
cloud network (cluster-subnet-ru-central1-a
subnet) with thecluster-vm-sg
security group.other-vm
: On thevm-net
cloud network (vm-subnet-ru-central1-a
subnet) with theother-vm-sg
security group.
To do this, create a configuration file called
vms.tf
describing the VMs.You also need to add the public parts of the SSH keys to this file to be able to connect to VMs via SSH.
vms.tf
resource "yandex_compute_disk" "boot-disk-1" { name = "boot-disk" type = "network-hdd" zone = "ru-central1-a" size = "20" image_id = fd81hgrcv6lsnkremf32 } resource "yandex_compute_disk" "boot-disk-2" { name = "boot-disk" type = "network-hdd" zone = "ru-central1-a" size = "20" image_id = fd81hgrcv6lsnkremf32 } resource "yandex_compute_instance" "cluster-vm" { name = "cluster-vm" platform_id = "standard-v3" zone = "ru-central1-a" resources { cores = 2 memory = 2 } boot_disk { disk_id = yandex_compute_disk.boot-disk-1.id } network_interface { subnet_id = yandex_vpc_subnet.cluster-subnet-a.id security_group_ids = [yandex_vpc_security_group.cluster-vm-sg.id] nat = true } metadata = { ssh-keys = "<any_username>:<SSH_key_contents>" } } resource "yandex_compute_instance" "other-vm" { name = "other-vm" platform_id = "standard-v3" zone = "ru-central1-a" resources { cores = 2 memory = 2 } boot_disk { disk_id = yandex_compute_disk.boot-disk-2.id } network_interface { subnet_id = yandex_vpc_subnet.vm-subnet-a.id security_group_ids = [yandex_vpc_default_security_group.other-vm-sg.id] nat = true } metadata = { ssh-keys = "<any_username>:<SSH_key_contents>" } }
For more information about creating a VM, see the Yandex Compute Cloud documentation.
-
Check the Terraform configuration files for errors:
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Apply your changes:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
Configure routing between the cloud networks
Configure routing between the cluster-net
and vm-net
cloud networks so that any subnet of one network is accessible from any subnet of the other network.
There are several ways to set up this routing. See the network routing tutorials. For example, you can create an IPsec tunnel.
Check cluster availability (optional step)
-
Connect to the
cluster-vm
andother-vm
virtual machines via SSH.Warning
-
If you created virtual machines with Ubuntu 20.04 LTS from the
fd81hgrcv6lsnkremf32
Yandex Cloud Marketplace image, useubuntu
as your username to connect. -
If you used a different image, the username may be different. For information on getting this name, see the Yandex Compute Cloud documentation.
-
-
Try connecting to the cluster:
-
From the
cluster-vm
virtual machine located in the same cloud network as the cluster.The connection will be successful.
-
From the
other-vm
virtual machine located in a different cloud network.The connection will fail because it will not be possible to get the IP address of the cluster host using the FQDN from another network. However, you can get the IP address of the cluster host and use this IP address to connect: this connection will be successful.
Next, we will demonstrate how to use shared DNS zone settings to maintain the ability to connect to a cluster from a different cloud network using FQDN.
-
Create a zone in Cloud DNS
To be able to resolve cluster FQDNs to IP addresses from other cloud networks, you need to create a DNS zone that will include these cloud networks.
Create an internal DNS zone called vpc-peering-zone
that will enable you to resolve cluster FQDNs from the vm-net
and cluster-net
cloud networks:
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan for the example.
-
Create a configuration file named
dns.tf
describing the DNS zone:dns.tf
resource "yandex_dns_zone" "vpc-peering-zone" { name = "vpc-peering-zone" zone = "mdb.yandexcloud.net." public = false private_networks = [yandex_vpc_network.vm-net.id, yandex_vpc_network.cluster-net.id] }
-
Check the Terraform configuration files for errors:
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Apply your changes:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
Make sure the DNS records migrate to the zone
DNS records for a cluster that existed before you created a DNS zone do not automatically migrate to this DNS zone.
For the cluster's DNS records to migrate to the new DNS zone, at least one DNS record must be updated in the internal zones of the cluster-net
and vm-net
cloud networks.
You can achieve this, for example, by creating a single virtual machine in each cloud network. You can delete these VMs after migrating the DNS records.
However, since you already created the сluster-vm
and other-vm
virtual machines, you only have to stop and then restart them.
When the migration process is complete, the cluster DNS records will appear in the vpc-peering-zone
DNS zone.
Tip
Before migrating the DNS records of production clusters, migrate the records in the test folder containing the test cluster. This will help ensure that migration occurs without errors.
Check cluster availability
-
Connect to the
cluster-vm
andother-vm
virtual machines via SSH.Warning
-
If you created virtual machines with Ubuntu 20.04 LTS from the
fd81hgrcv6lsnkremf32
Yandex Cloud Marketplace image, useubuntu
as your username to connect. -
If you used a different image, the username may be different. For information on getting this name, see the Yandex Compute Cloud documentation.
-
-
Try connecting to the cluster:
- From the virtual machine
cluster-vm
located in the same cloud network as the cluster. - From the
other-vm
virtual machine located in a different cloud network.
Both connections should be successful.
- From the virtual machine
Delete the resources you created
If you no longer need the resources you created, delete them:
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan for the example.
-
Run this command:
terraform destroy
-
Type
yes
and press Enter.
ClickHouse® is a registered trademark of ClickHouse, Inc