Contact UsGet started

Configuring Yandex Cloud DNS for Yandex Managed Service for ClickHouse® cluster access from other cloud networks

Written by
Updated at October 1, 2024

Clients (e.g., virtual machines) in the same cloud network as the cluster can access the cluster's hosts using their fully qualified domain names (FQDNs). However, if the clients are in another cloud network, this will not be possible. This is because the required DNS service zones are created automatically and only support name resolution within a single cloud network.

Yandex Cloud DNS enables you to configure a DNS zone shared by multiple cloud networks. This makes it possible to access cluster hosts by their FQDNs even if the client and the cluster are on different cloud networks.

Warning

Cluster hosts with public access enabled will become inaccessible by their FQDNs from within the cloud networks with the shared DNS zone. However, such hosts will still be accessible by their FQDNs from the internet.

To set up a shared DNS zone:

  1. Create a zone in Cloud DNS.
  2. Make sure the DNS records migrate to the zone.
  3. Check cluster availability.

If you no longer need the resources you created, delete them.

The following is an example where:

  1. You place a cluster and a VM named cluster-vm in a cloud network named cluster-net.
  2. You create a new cloud network named vm-net with a VM named other-vm.
  3. You check the cluster for accessibility from these VMs.
  4. You create a DNS zone named vpc-peering-zone and perform a migration.
  5. You check the cluster for accessibility from these VMs once again.

Getting started

Assign roles

Make sure you have access to cloud networks in the folders where the cluster-net and vm-net networks will reside.

You must have one of the following roles:

  • admin
  • editor
  • vpc.admin

For information on assigning roles, see the Yandex Identity and Access Management documentation.

Create a cluster

  1. Using the command line, navigate to the folder that will contain the Terraform configuration files with an infrastructure plan for the example. Create the directory if it does not exist.
  2. Create a multi-host Managed Service for ClickHouse® cluster using Terraform by following the example.

Create other resources

  1. Describe the new vm-net cloud network with the vm-subnet-ru-central1-a subnet: CIDR 172.17.1.0/24, ru-central1-a availability zone. To do this, add the following lines to the networks.tfconfiguration file:

    networks.tf
    resource "yandex_vpc_network" "vm-net" { name = "vm-net" }

resource "yandex_vpc_subnet" "vm-subnet-a" {
  name           = "vm-subnet-ru-central1-a"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.vm-net.id
  v4_cidr_blocks = ["172.17.1.0/24"]
}

  2. Describe the security groups:

    • Additional security group named cluster-vm-sg in the cluster-net network for the cluster-vm virtual machine.
    • Default security group named other-vm-sg in the vm-net network for the other-vm virtual machine.

    To do this, add the following lines to the security-groups.tf configuration file:

    security-groups.tf
    resource "yandex_vpc_security_group" "cluster-vm-sg" {
  name       = "cluster-vm-sg"
  network_id = yandex_vpc_network.cluster-net.id

  ingress {
    description    = "Allow any ingress traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description    = "Allow any egress traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "yandex_vpc_default_security_group" "other-vm-sg" {
  network_id = yandex_vpc_network.vm-net.id

  ingress {
    description    = "Allow any ingress traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description    = "Allow any egress traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

    Warning

    The above security groups allow any traffic in any direction and serve for demonstration purposes. This enables you to install the utilities required for connection on the VM and to connect to the cluster.

    For real-world tasks, configure the security groups to allow only essential traffic.

  3. Describe the VMs to use for migration and DNS functionality check:

    • cluster-vm: On the cluster-net cloud network (cluster-subnet-ru-central1-a subnet) with the cluster-vm-sg security group.
    • other-vm: On the vm-net cloud network (vm-subnet-ru-central1-a subnet) with the other-vm-sg security group.

    To do this, create a configuration file called vms.tf describing the VMs.

    You also need to add the public parts of the SSH keys to this file to be able to connect to VMs via SSH.

    vms.tf
    resource "yandex_compute_disk" "boot-disk-1" {
  name     = "boot-disk"
  type     = "network-hdd"
  zone     = "ru-central1-a"
  size     = "20"
  image_id = fd81hgrcv6lsnkremf32
}

resource "yandex_compute_disk" "boot-disk-2" {
  name     = "boot-disk"
  type     = "network-hdd"
  zone     = "ru-central1-a"
  size     = "20"
  image_id = fd81hgrcv6lsnkremf32
}

resource "yandex_compute_instance" "cluster-vm" {
  name        = "cluster-vm"
  platform_id = "standard-v3"
  zone        = "ru-central1-a"

  resources {
    cores  = 2
    memory = 2
  }

  boot_disk {
    disk_id = yandex_compute_disk.boot-disk-1.id
  }

  network_interface {
    subnet_id          = yandex_vpc_subnet.cluster-subnet-a.id
    security_group_ids = [yandex_vpc_security_group.cluster-vm-sg.id]
    nat                = true
  }

  metadata = {
    ssh-keys = "<any_username>:<SSH_key_contents>"
  }
}

resource "yandex_compute_instance" "other-vm" {
  name        = "other-vm"
  platform_id = "standard-v3"
  zone        = "ru-central1-a"

  resources {
    cores  = 2
    memory = 2
  }

  boot_disk {
    disk_id = yandex_compute_disk.boot-disk-2.id
  }

  network_interface {
    subnet_id          = yandex_vpc_subnet.vm-subnet-a.id
    security_group_ids = [yandex_vpc_default_security_group.other-vm-sg.id]
    nat                = true
  }

  metadata = {
    ssh-keys = "<any_username>:<SSH_key_contents>"
  }
}

    For more information about creating a VM, see the Yandex Compute Cloud documentation.

  4. Check the Terraform configuration files for errors:

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Apply your changes:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Configure routing between the cloud networks

Configure routing between the cluster-net and vm-net cloud networks so that any subnet of one network is accessible from any subnet of the other network.

There are several ways to set up this routing. See the network routing tutorials. For example, you can create an IPsec tunnel.

Check cluster availability (optional step)

  1. Connect to the cluster-vm and other-vm virtual machines via SSH.

    Warning

  2. Try connecting to the cluster:

    • From the cluster-vm virtual machine located in the same cloud network as the cluster.

      The connection will be successful.

    • From the other-vm virtual machine located in a different cloud network.

      The connection will fail because it will not be possible to get the IP address of the cluster host using the FQDN from another network. However, you can get the IP address of the cluster host and use this IP address to connect: this connection will be successful.

      Next, we will demonstrate how to use shared DNS zone settings to maintain the ability to connect to a cluster from a different cloud network using FQDN.

Create a zone in Cloud DNS

To be able to resolve cluster FQDNs to IP addresses from other cloud networks, you need to create a DNS zone that will include these cloud networks.

Create an internal DNS zone called vpc-peering-zone that will enable you to resolve cluster FQDNs from the vm-net and cluster-net cloud networks:

  1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan for the example.

  2. Create a configuration file named dns.tf describing the DNS zone:

    dns.tf
    resource "yandex_dns_zone" "vpc-peering-zone" {
  name             = "vpc-peering-zone"
  zone             = "mdb.yandexcloud.net."
  public           = false
  private_networks = [yandex_vpc_network.vm-net.id, yandex_vpc_network.cluster-net.id]
}

  3. Check the Terraform configuration files for errors:

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Apply your changes:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Make sure the DNS records migrate to the zone

DNS records for a cluster that existed before you created a DNS zone do not automatically migrate to this DNS zone.

For the cluster's DNS records to migrate to the new DNS zone, at least one DNS record must be updated in the internal zones of the cluster-net and vm-net cloud networks.

You can achieve this, for example, by creating a single virtual machine in each cloud network. You can delete these VMs after migrating the DNS records.

However, since you already created the сluster-vm and other-vm virtual machines, you only have to stop and then restart them.

When the migration process is complete, the cluster DNS records will appear in the vpc-peering-zone DNS zone.

Tip

Before migrating the DNS records of production clusters, migrate the records in the test folder containing the test cluster. This will help ensure that migration occurs without errors.

Check cluster availability

  1. Connect to the cluster-vm and other-vm virtual machines via SSH.

    Warning

  2. Try connecting to the cluster:

    • From the virtual machine cluster-vm located in the same cloud network as the cluster.
    • From the other-vm virtual machine located in a different cloud network.

    Both connections should be successful.

Delete the resources you created

If you no longer need the resources you created, delete them:

  1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan for the example.

  2. Run this command:

    terraform destroy

  3. Type yes and press Enter.

ClickHouse® is a registered trademark of ClickHouse, Inc.

Previous
Exchanging data with Yandex Data Processing
Next
Analyzing Yandex Object Storage logs in Yandex DataLens