Networking in Yandex Data Processing
All subclusters of a cluster belong to the same cloud network, while all hosts of each subcluster are in a certain subnet of that network.
When creating a cluster, public access can be enabled for any subcluster to make its hosts available online. You can only connect to subcluster hosts without public access through Yandex Cloud VMs located in the same cloud network as the cluster.
Cluster host addresses
When you create a host in a subcluster, Yandex Data Processing generates an FQDN and IP address for it. You can use them to access the host within a single cloud network.
The host IP address may change during operation; however, its FQDN is permanent.
To learn how to get a host FQDN, see this guide.
Warning
When you reduce the number of hosts in a subcluster, it is Yandex Data Processing that selects the hosts to remove. The FQDNs of the removed hosts stop working.
Assigning network aliases to hosts
To maintain external network access to Yandex Data Processing services, create a network alias (CNAME record) in Yandex Cloud DNS to point to the relevant name of the Yandex Data Processing cluster master host.
To reconfigure external connections when recreating a cluster or moving the workload to a different cluster, you can just change the CNAME record you created.
For a configuration example, see the Reconfiguring a network connection when recreating a cluster section.
Security groups
Security groups follow the All traffic that is not allowed is prohibited principle. If the security group settings are missing the required rules, you will not be able to connect to the cluster. Furthermore, there will be no connectivity between the subclusters, the cluster, and the intermediate VM instance used for port forwarding.
For example, let's assume you use a VM located on the 10.128.0.0/16 subnet to connect to the cluster. If only the 10.133.0.0/24 subnet is specified in the security group rules, you will not be able to connect to the cluster. Moreover, you will not be able to connect to a cluster with a VM located in the 10.128.0.0/16 subnet, for which the permissions for the required ports have not been specified.
Before creating a cluster, you should create and configure security groups so that service traffic between cluster hosts is enabled. For more information, see Creating a cluster.