Setting up NodeLocal DNS Cache

Written by

Updated at September 3, 2024

Getting started
- Create an infrastructure
- Prepare the environment
Install NodeLocal DNS
Change the NodeLocal DNS Cache configuration
Run DNS queries
Set up traffic through NodeLocal DNS
Check logs
Stop the DaemonSet
Delete the resources you created

To reduce the number of DNS queries on a Managed Service for Kubernetes cluster, enable NodeLocal DNS Cache.

Tip

If a Managed Service for Kubernetes cluster is made up of over 50 nodes, use automatic DNS scaling.

By default, pods send queries to the kube-dns service. The nameserver field in the /etc/resolv.conf file is set to the ClusterIp value of kube-dns. A connection to the ClusterIP is established using iptables or IP Virtual Server.

When NodeLocal DNS Cache is enabled, a DaemonSet is deployed in a Managed Service for Kubernetes cluster. The caching agent is run on each Managed Service for Kubernetes node (under node-local-dns). User pods now send queries to the agent running on their Managed Service for Kubernetes nodes.

If a query is in the agent cache, it returns a direct response. Otherwise, a TCP connection to the ClusterIP kube-dns is created. By default, the caching agent makes cache-miss requests to kube-dns for the cluster.local DNS zone of the Managed Service for Kubernetes cluster.

This helps avoid the DNAT rules, connection tracking, and restrictions on the number of connections. For more information about NodeLocal DNS Cache, see the documentation.

To set up DNS query caching:

Getting started

Create an infrastructure

Manually

Terraform

Create a cloud network and subnet.
Create a service account with the editor role.
Create security groups for the Managed Service for Kubernetes cluster and its node groups.

Warning

The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
Create a Managed Service for Kubernetes cluster and a node group with public internet access and the security groups prepared earlier.

If you do not have Terraform yet, install it.
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
Download the k8s-node-local-dns.tf configuration file of the Managed Service for Kubernetes cluster to the same working directory. The file describes:
- Network.
- Subnet.
- Managed Service for Kubernetes cluster.
- Service account required for the cluster and Managed Service for Kubernetes node group to operate.
- Security groups which contain rules required for the Managed Service for Kubernetes cluster and its node groups.
  
  Warning
  
  The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
Specify the following in the configuration file:
- Folder ID.
- Kubernetes versions for the cluster and Managed Service for Kubernetes node groups.
- Managed Service for Kubernetes cluster CIDR.
- Name of the Managed Service for Kubernetes cluster service account.
Make sure the Terraform configuration files are correct using this command:
```
terraform validate
```
If there are any errors in the configuration files, Terraform will point them out.
Create the required infrastructure:
1. Run the command to view planned changes:
```
terraform plan
```
  If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
2. If you are happy with the planned changes, apply them:
  1. Run the command:
```
terraform apply
```
  2. Confirm the update of resources.
  3. Wait for the operation to complete.
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Prepare the environment

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.
Install kubectl and configure it to work with the created cluster.

Install NodeLocal DNS

Yandex Cloud Marketplace

Manually

Install NodeLocal DNS using Cloud Marketplace as described in this guide.

Retrieve the service IP address for kube-dns:

kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}

Create a file named node-local-dns.yaml. In the node-local-dns DaemonSet settings, specify the kube-dns service IP address:

node-local-dns.yaml

# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Modified for Yandex Cloud Usage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns-upstream
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "KubeDNSUpstream"
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: kube-dns
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
data:
  Corefile: |
    cluster.local:53 {
      errors
      cache {
        success 9984 30
        denial 9984 5
      }
      reload
      loop
      bind 169.254.20.10 <kube-dns_IP_address>
      forward . __PILLAR__CLUSTER__DNS__ {
        prefer_udp
      }
      prometheus :9253
      health 169.254.20.10:8080
    }
    in-addr.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.20.10 <kube-dns_IP_address>
      forward . __PILLAR__CLUSTER__DNS__ {
        prefer_udp
      }
      prometheus :9253
      }
    ip6.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.20.10 <kube-dns_IP_address>
      forward . __PILLAR__CLUSTER__DNS__ {
        prefer_udp
      }
      prometheus :9253
      }
    .:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.20.10 <kube-dns_IP_address>
      forward . __PILLAR__UPSTREAM__SERVERS__ {
        prefer_udp
      }
      prometheus :9253
      }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: true
      dnsPolicy: Default # Don't use cluster DNS.
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - effect: "NoExecute"
        operator: "Exists"
      - effect: "NoSchedule"
        operator: "Exists"
      containers:
      - name: node-cache
        image: registry.k8s.io/dns/k8s-dns-node-cache:1.17.0
        resources:
          requests:
            cpu: 25m
            memory: 5Mi
        args: [ "-localip", "169.254.20.10,<kube-dns_IP_address>", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream" ]
        securityContext:
          privileged: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9253
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            host: 169.254.20.10
            path: /health
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - name: config-volume
          mountPath: /etc/coredns
        - name: kube-dns-config
          mountPath: /etc/kube-dns
      volumes:
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      - name: config-volume
        configMap:
          name: node-local-dns
          items:
            - key: Corefile
              path: Corefile.base
---
# A headless service is a service with a service IP but instead of load-balancing it will return the IPs of our associated Pods.
# We use this to expose metrics to Prometheus.
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9253"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-local-dns
  name: node-local-dns
  namespace: kube-system
spec:
  clusterIP: None
  ports:
    - name: metrics
      port: 9253
      targetPort: 9253
  selector:
    k8s-app: node-local-dns

Create resources for NodeLocal DNS:

kubectl apply -f node-local-dns.yaml

Result:

serviceaccount/node-local-dns created
service/kube-dns-upstream created
configmap/node-local-dns created
daemonset.apps/node-local-dns created
service/node-local-dns created

Make sure that DaemonSet is successfully deployed and running:

kubectl get ds -l k8s-app=node-local-dns -n kube-system

Result:

NAME            DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
node-local-dns  3        3        3      3           3          <none>         24m

Change the NodeLocal DNS Cache configuration

To change the configuration, edit the appropriate configmap. For example, to enable DNS request logging for the cluster.local zone:

Run this command:

kubectl -n kube-system edit configmap node-local-dns

Add the log line to the cluster.local zone configuration:

...
apiVersion: v1
  data:
    Corefile: |
      cluster.local:53 {
          log
          errors
          cache {
                  success 9984 30
                  denial 9984 5
          }
...

Save your changes.

Result:
```
configmap/node-local-dns edited
```

It may take several minutes to update the configuration.

Run DNS queries

To run test queries, use the pod with the DNS diagnostic utilities.

Run the pod:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

Result:

pod/dnsutils created

Make sure the pod status changed to Running:

kubectl get pods dnsutils

Result:

NAME      READY  STATUS   RESTARTS  AGE
dnsutils  1/1    Running  0         26m

Connect to a pod:
```
kubectl exec -i -t dnsutils -- sh
```

Get the IP address of the local DNS cache:

nslookup kubernetes.default

Result:

Server:         <kube-dns_IP_address>
Address:        <kube-dns_IP_address>#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.128.1

Run the following queries:
```
dig +short @169.254.20.10 www.com
dig +short @<kube-dns_IP_address> example.com
```
Result:
```
# dig +short @169.254.20.10 www.com
52.128.23.153
# dig +short @<kube-dns_IP_address> example.com
93.184.216.34
```
After node-local-dns launches, the iptables rules will be configured for the local DNS to respond on both of the addresses (<kube-dns_IP>:53 and 169.254.20.10:53).

The kube-dns service can be accessed at the ClusterIp address of kube-dns-upstream. You may need this address to configure request forwarding.

Set up traffic through NodeLocal DNS

All pods

Selected pods

Create a pod for network traffic setup:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: dnschange
  namespace: default
spec:
  priorityClassName: system-node-critical
  hostNetwork: true
  dnsPolicy: Default
  hostPID: true
  tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - effect: "NoExecute"
      operator: "Exists"
    - effect: "NoSchedule"
      operator: "Exists"
  containers:
  - name: dnschange
    image: registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3
    tty: true
    stdin: true
    securityContext:
      privileged: true
    command:
      - nsenter
      - --target
      - "1"
      - --mount
      - --uts
      - --ipc
      - --net
      - --pid
      - --
      - sleep
      - "infinity"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

Connect to the dnschange pod you created:
```
kubectl exec -it dnschange -- sh
```
Open the /etc/default/kubelet file in the container to edit it:
```
vi /etc/default/kubelet
```

In the file, add the --cluster-dns=169.254.20.10 parameter (NodeLocal DNS cache address) to the KUBELET_OPTS variable value:

KUBELET_OPTS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubeconfig.conf --cert-dir=/var/lib/kubelet/pki/   --cloud-provider=external --config=/home/kubernetes/kubelet-config.yaml --kubeconfig=/etc/kubernetes/  kubelet-kubeconfig.conf --resolv-conf=/run/systemd/resolve/resolv.conf --v=2 --cluster-dns=169.254.20.10"

Save the file and run the kubelet restart command:
```
systemctl daemon-reload && systemctl restart kubelet
```
Next, exit the container mode by running the exit command.
Delete the dnschange pod:
```
kubectl delete pod dnschange
```

To make sure all pods start running through NodeLocal DNS, restart them, e.g., using the command below:

kubectl get deployments --all-namespaces | \
  tail +2 | \
  awk '{
    cmd=sprintf("kubectl rollout restart deployment -n %s %s", $1, $2) ;
    system(cmd)
  }'

Run this command:

kubectl edit deployment <pod_deployment_name>

In the pod specification, replace the dnsPolicy: ClusterFirst setting in the spec.template.spec key with the following section:

  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 169.254.20.10
    searches:
      - default.svc.cluster.local
      - svc.cluster.local
      - cluster.local
      - ru-central1.internal
      - internal
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "5"

Check logs

Run this command:

kubectl logs --namespace=kube-system -l k8s-app=node-local-dns -f

To stop displaying a log, press Ctrl + C.

Result:

...
[INFO] 10.112.128.7:50527 - 41658 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000097538s
[INFO] 10.112.128.7:44256 - 26847 "AAAA IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 147 0.057075876s
...

Stop the DaemonSet

To disable DaemonSet in NodeLocal DNS Cache, run:

kubectl delete -f node-local-dns.yaml

Result:

serviceaccount "node-local-dns" deleted
service "kube-dns-upstream" deleted
configmap "node-local-dns" deleted
daemonset.apps "node-local-dns" deleted
service "node-local-dns" deleted

Delete the resources you created

Delete the resources you no longer need to avoid paying for them:

Manually

Terraform

Delete the Managed Service for Kubernetes cluster.
If static public IP addresses were used for Managed Service for Kubernetes cluster and node access, release and delete them.

In the command line, go to the directory with the current Terraform configuration file with an infrastructure plan.
Delete the k8s-node-local-dns.tf configuration file.
Make sure the Terraform configuration files are correct using this command:
```
terraform validate
```
If there are any errors in the configuration files, Terraform will point them out.
Confirm updating the resources.
1. Run the command to view planned changes:
```
terraform plan
```
  If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
2. If you are happy with the planned changes, apply them:
  1. Run the command:
```
terraform apply
```
  2. Confirm the update of resources.
  3. Wait for the operation to complete.
All the resources described in the k8s-node-local-dns.tf configuration file will be deleted.
If static public IP addresses were used for Managed Service for Kubernetes cluster and node access, release and delete them.

Setting up NodeLocal DNS Cache

Getting startedGetting started

Create an infrastructureCreate an infrastructure

Prepare the environmentPrepare the environment

Install NodeLocal DNSInstall NodeLocal DNS

Change the NodeLocal DNS Cache configurationChange the NodeLocal DNS Cache configuration

Run DNS queriesRun DNS queries

Set up traffic through NodeLocal DNSSet up traffic through NodeLocal DNS

Check logsCheck logs

Stop the DaemonSetStop the DaemonSet

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Create an infrastructure

Prepare the environment

Install NodeLocal DNS

Change the NodeLocal DNS Cache configuration

Run DNS queries

Set up traffic through NodeLocal DNS

Check logs

Stop the DaemonSet

Delete the resources you created