Configuring masquerading in clusters with multiple ranges of pod IP addresses

Create a cluster with multiple ranges of pod IP addresses and expand the address space of the existing cluster. Then, check the masquerading settings.

The ConfigMap object named ip-masq-agent in the kube-system namespace contains the settings for pod IP address masquerading. The data.config.nonMasqueradeCIDRs sets destination ranges, which, when accessed, do not get the pod IP address to be replaced with the node IP address.

In clusters with multiple ranges of pod IP addresses, save all the cluster's pod ranges in this list.

Getting startedGetting started

You can specify the ranges when creating a cluster or add them later when updating the cluster.

View and save the masquerading settings before adding a range to an existing cluster. If you use your own ranges or other settings, enable manual management to save them.

Synchronize masquerading after adding a rangeSynchronize masquerading after adding a range

1. Check the management mode for ip-masq-agent ConfigMap.
2. Based on the mode, follow these steps:
   • enforced: Wait for the next configuration update and check that the new range is in nonMasqueradeCIDRs. Do not edit data.config manually: the service will overwrite the changes.
   • disabled: Run kubectl -n kube-system edit configmap ip-masq-agent and add the new range to data.config.nonMasqueradeCIDRs. Save the previous ranges and other settings. For a configuration example, see this section.
   • No annotation: Wait for automatic mode selection and read ConfigMap again. If you need to save the custom settings, explicitly specify disabled before editing them.
3. Check the result.

Management modesManagement modes

The mode is specified by the addon-cm-controller.mk8s.yc.io/mode annotation:

The configuration is checked periodically: every 60 seconds by default. Changing mode does not require restarting the cluster or nodes.

View the current settingsView the current settings

First, install kubectl and configure it to work with the Managed Service for Kubernetes cluster you created.

Run this command to view the mode and ConfigMap contents:

kubectl -n kube-system get configmap ip-masq-agent -o yaml

Before editing, save the current configuration:

kubectl -n kube-system get configmap ip-masq-agent -o yaml > ip-masq-agent-backup.yaml

Enable automatic managementEnable automatic management

Use enforced mode if you have enough pod system ranges and standard ip-masq-agent settings.

Run this command to delegate management to the service:

kubectl -n kube-system annotate configmap ip-masq-agent \
  addon-cm-controller.mk8s.yc.io/mode=enforced --overwrite

When adding a new pod range, the service will automatically add it to nonMasqueradeCIDRs.

Enable manual managementEnable manual management

If you are adding custom ranges or editing other ip-masq-agent settings:

1. Enable disabled mode:

   kubectl -n kube-system annotate configmap ip-masq-agent \
     addon-cm-controller.mk8s.yc.io/mode=disabled --overwrite
   

2. Open ConfigMap to edit it:

   kubectl -n kube-system edit configmap ip-masq-agent
   

3. In nonMasqueradeCIDRs, save all the cluster's pod ranges and add custom destination ranges that do not require masquerading. Here is an example:

   data:
     config: |+
       nonMasqueradeCIDRs:
         - 10.112.0.0/16
         - 10.113.0.0/16
         - 10.200.0.0/16
       masqLinkLocal: false
       resyncInterval: 60s
   

   In the example, the first two ranges belong to pods, and the third, to the custom network. Replace them with your values. Save the other settings required for your network.

Automatic mode selectionAutomatic mode selection

At the first run, the controller checks ConfigMap without any annotation. If the configuration only lists the expected system ranges, enforced mode is enabled. A configuration with additional, missing, or modified ranges is assigned disabled mode and saved without any changes.

Comparisons factor in the range order. IPv4 system ranges in the previous format that spanned half of the range may be recognized and replaced with full ones. A combined list of full and half ranges is considered a custom configuration.

Deleting the annotation triggers repeat automatic selection. To select an exact mode, explicitly specify enforced or disabled.

Check the resultCheck the result

1. After creating new nodes, check the address blocks assigned to them:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,POD_CIDR:.spec.podCIDR,POD_CIDRS:.spec.podCIDRs
   

   The new range may not be used if the previous one still has unused blocks. When adding a range, the IP addresses of existing nodes and pods do not change.

2. Wait for the next configuration check and view the settings:

   kubectl -n kube-system get configmap ip-masq-agent -o yaml
   

3. Check the ConfigMap events:

   kubectl -n kube-system get events \
     --field-selector involvedObject.kind=ConfigMap,involvedObject.name=ip-masq-agent \
     --sort-by=.lastTimestamp
   

   If you see an InvalidMode event, fix the annotation value.

4. To check the node masquerading rules, connect to that node over SSH and run this command:

   sudo iptables -t nat -L IP-MASQ -v -n
   

Do not delete ConfigMap to reset the settings, as the service will recreate it using the system template. Use mode annotation to change the behavior.

Useful linksUseful links

• Clusters with multiple ranges of pod IP addresses
• Networking in Managed Service for Kubernetes