Cluster Interconnection (Alpha)#

It supports configuration of cluster interconnection between clusters whose network mode is the same as Kube-OVN, so that Pods in the clusters can access each other. Cluster Interconnect Controller is an extension component provided by Kube-OVN, which is responsible for collecting network information between different clusters and connecting the networks of multiple clusters by issuing routes.

#Prerequisites

• The subnet CIDRs of different clusters cannot overlap each other.

• There needs to be a set of machines that can be accessed over IP by each cluster's kube-ovn-controller to deploy controllers that interconnect across clusters.

• A set of machines that can be accessed by kube-ovn-controller per cluster via IP for cross-cluster interconnections needs to exist for each cluster to be used as gateway nodes afterward.

• This feature is only available for the default VPC, user-defined VPCs cannot use the interconnect feature.

#Multi-node Kube-OVN connectivity controller was built

There are three deployment methods available: Deploy deployment (supported in platform v3.16.0 and later versions), Docker deployment, and Containerd deployment.

#Deploy Deployment

Note: This deployment method is supported in platform v3.16.0 and later versions.

Operation Steps

1. Execute the following command on the cluster Master node to obtain the install-ic-server.sh installation script.

   wget https://github.com/kubeovn/kube-ovn/blob/release-1.12/dist/images/install-ic-server.sh

2. Open the script file in the current directory and modify the parameters as follows.

   REGISTRY="kubeovn"
   VERSION=""

   Modified parameter configurations are as follows:

   REGISTRY="<Kube-OVN image repository address>"   ## For example: REGISTRY="registry.alauda.cn:60080/acp/"
   VERSION="<Kube-OVN version>"   ## For example: VERSION="v1.9.25"

3. Save the script file and execute it using the following command.

   sh install-ic-server.sh

#Docker and Containerd Deployment

1. Select three or more nodes in any cluster to deploy the Interconnected Controller. In this example, three nodes are prepared.

2. Choose any node as the Leader and execute the following commands according to the different deployment methods.

   Note: Before configuration, please check if there is an ovn directory under /etc. If not, use the command mkdir /etc/ovn to create one.

   • Commands for Docker deployment

     Note: Execute the command docker images | grep ovn to obtain the Kube-OVN image address.

     • Command for the Leader node:

       docker run \
       --name=ovn-ic-db \
       -d \
       --env "ENABLE_OVN_LEADER_CHECK=false" \
       --network=host \
       --restart=always \
       --privileged=true \
       -v /etc/ovn/:/etc/ovn \
       -v /var/run/ovn:/var/run/ovn \
       -v /var/log/ovn:/var/log/ovn \
       -e LOCAL_IP="<IP address of the current node>" \   ## For example: -e LOCAL_IP="192.168.39.37"
       -e NODE_IPS="<IP addresses of all nodes, separated by commas>" \   ## For example: -e NODE_IPS="192.168.39.22,192.168.39.24,192.168.39.37"
       <image repository address> bash start-ic-db.sh    ## For example: 192.168.39.10:60080/acp/kube-ovn:v1.8.8 bash start-ic-db.sh

     • Commands for the other two nodes:

       docker run \
       --name=ovn-ic-db \
       -d \
       --env "ENABLE_OVN_LEADER_CHECK=false" \
       --network=host \
       --restart=always \
       --privileged=true \
       -v /etc/ovn/:/etc/ovn \
       -v /var/run/ovn:/var/run/ovn \
       -v /var/log/ovn:/var/log/ovn \
       -e LOCAL_IP="<IP address of the current node>" \   ## For example: -e LOCAL_IP="192.168.39.24"
       -e LEADER_IP="<IP address of the Leader node>" \  ## For example: -e LEADER_IP="192.168.39.37"
       -e NODE_IPS="<IP addresses of all nodes, separated by commas>" \   ## For example: -e NODE_IPS="192.168.39.22,192.168.39.24,192.168.39.37"
       <image repository address> bash start-ic-db.sh   ## For example: 192.168.39.10:60080/acp/kube-ovn:v1.8.8  bash start-ic-db.sh

   • Commands for Containerd deployment

     Note: Execute the command crictl images | grep ovn to obtain the Kube-OVN image address.

     • Command for the Leader node:

       ctr -n k8s.io run \
       -d \
       --env "ENABLE_OVN_LEADER_CHECK=false" \
       --net-host \
       --privileged \
       --mount="type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw" \
       --mount="type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw" \
       --mount="type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw" \
       --env="NODE_IPS=<IP addresses of all nodes, separated by commas>" \   ## For example: --env="NODE_IPS="192.168.178.97,192.168.181.93,192.168.177.192""
       --env="LOCAL_IP=<IP address of the current node>" \   ## For example: --env="LOCAL_IP="192.168.178.97""
       <image repository address> ovn-ic-db bash start-ic-db.sh   ## For example: registry.alauda.cn:60080/acp/kube-ovn:v1.9.25 ovn-ic-db bash start-ic-db.sh

     • Commands for the other two nodes:

       ctr -n k8s.io run \
       -d \
       --env "ENABLE_OVN_LEADER_CHECK=false" \
       --net-host \
       --privileged \
       --mount="type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw" \
       --mount="type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw" \
       --mount="type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw" \
       --env="NODE_IPS=<IP addresses of all nodes, separated by commas>" \   ## For example: --env="NODE_IPS="192.168.178.97,192.168.181.93,192.168.177.192"" \
       --env="LOCAL_IP=<IP address of the current node>" \   ## For example: --env="LOCAL_IP="192.168.181.93""
       --env="LEADER_IP=<IP address of the Leader node>" \   ## For example: --env="LEADER_IP="192.168.178.97""
       <image repository address> ovn-ic-db bash start-ic-db.sh   ## For example: registry.alauda.cn:60080/acp/kube-ovn:v1.9.25 ovn-ic-db bash start-ic-db.sh

#Deploy the cluster interconnection controller in the Global cluster

In any control node of global, replace the following parameters according to the comments and execute the following command to create the ConfigMap resource.

Note: To ensure the correct operation, the ConfigMap named ovn-ic on global is not allowed to be modified. If any parameter needs to be changed, please delete the ConfigMap and reconfigure it correctly before applying the ConfigMap.

cat << EOF |kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: ovn-ic
  namespace: cpaas-system
data:
  ic-db-host: "192.168.39.22,192.168.39.24,192.168.39.37"   # Address of the node where the cluster interconnect controller is located, in this case, the local IP of the three nodes where the controller is deployed
  ic-nb-port: "6645"            # Cluster Interconnect Controller nb port, default 6645
  ic-sb-port: "6646"            # Cluster Interconnect Controller sb port, default 6646
EOF

#Join the cluster interconnect

Add a cluster whose network mode is Kube-OVN to the cluster interconnect.

Prerequisites

The created subnets, ovn-default, and join subnets in a cluster do not conflict with any cluster segment in the cluster interconnection group.

Procedure of operation

1. In the left navigation bar, click Clusters > Cluster of clusters.

2. Click the name of the cluster to be added to the cluster interconnect.

3. In the upper right corner, click Options > Cluster Interconnect.

4. Click Join the cluster interconnect.

5. Select a gateway node for the cluster.

6. Click Join.

#Relevant operations

#Update the gateway node information of the interconnected cluster

Update information about cluster gateway nodes that have joined a cluster interconnect group.

Procedure of operation

1. In the left navigation bar, click Clusters > Cluster of clusters.

2. Click Cluster name for the gateway node information to be updated.

3. In the upper-right corner, click Operations > Cluster Interconnect.

4. Click Update Gateway Node for the cluster whose gateway node information you want to update.

5. Reselect the gateway node for the cluster.

6. Click Update.

#Exit cluster interconnection

A cluster that has joined a cluster interconnection group exits cluster interconnection, and when it does, it disconnects the cluster Pod from the external cluster Pod.

Procedure of operation

1. In the left navigation bar, click Clusters > Cluster of clusters.

2. Click the name of the cluster that you want to decommission.

3. In the upper-right corner, click Options > Cluster Interconnect.

4. Click Exit cluster interconnection for the cluster you want to exit.

5. Enter the cluster name correctly.

6. Click Exit.

#Cleaning up Interconnected Cluster Residue

When a cluster is deleted without leaving the interconnected cluster, some residual data may remain on the controller. When you attempt to use these nodes to create a cluster again and join the interconnected cluster, failures may occur. You can check the detailed error information in the /var/log/ovn/ovn-ic.log log of the controller (kube-ovn-controller). Some error messages may include:

transaction error: {"details":"Transaction causes multiple rows in xxxxxx"}

Operational Steps

1. Exit the interconnected cluster for the cluster to be joined.

2. Execute the cleanup script in the container or pod.

   You can execute the cleanup script directly in either the ovn-ic-db container or the ovn-ic-controller pod. Choose one of the following methods:

   Method 1: Execute in ovn-ic-db container

   • Enter the ovn-ic-db container and perform the cleanup operation with the following commands.

     ctr -n k8s.io task exec -t --exec-id ovn-ic-db ovn-ic-db /bin/bash

     Then execute one of the following cleanup commands:

     • Execute the cleanup operation with the name of the original cluster. Replace <cluster-name> with the name of the original cluster:

       ./clean-ic-az-db.sh <cluster-name>

     • Execute the cleanup operation with the name of any node in the original cluster. Replace <node-name> with the name of any node in the original cluster:

       ./clean-ic-az-db.sh <node-name>

   Method 2: Execute in ovn-ic-controller pod

   • Enter the ovn-ic-controller pod and perform the cleanup operation with the following commands.

     kubectl -n kube-system exec -ti $(kubectl get pods -n kube-system -l app=ovn-ic-controller -o custom-columns=NAME:.metadata.name --no-headers) -- /bin/bash

     Then execute one of the following cleanup commands:

     • Execute the cleanup operation with the name of the original cluster. Replace <cluster-name> with the name of the original cluster:

       ./clean-ic-az-db.sh <cluster-name>

     • Execute the cleanup operation with the name of any node in the original cluster. Replace <node-name> with the name of any node in the original cluster:

       ./clean-ic-az-db.sh <node-name>

#Uninstalling the Interconnected Cluster

Note: Step 1 to Step 3 need to be performed on all business clusters that have joined the interconnected cluster.

Operational Steps

1. Exit the interconnected cluster. There are two specific exit methods, choose one according to your needs.

   • Delete the ConfigMap named ovn-ic-config in the business cluster. Use the following command.

     kubectl -n kube-system delete cm ovn-ic-config

   • Exit the interconnected cluster through platform operations.

2. Enter the Leader Pod of ovn-central with the following command.

   kubectl -n kube-system exec -ti $(kubectl get pods -n kube-system -lovn-nb-leader=true -o custom-columns=NAME:.metadata.name --no-headers) -- /bin/bash

3. Clean up the ts logical switch with the following command.

   ovn-nbctl ls-del ts

4. Log in to the node where the controller is deployed and delete the controller.

   • Docker command:

     docker stop ovn-ic-db
     docker rm ovn-ic-db

   • Containerd command:

     ctr -n k8s.io task kill ovn-ic-db
     ctr -n k8s.io containers rm ovn-ic-db

5. Delete the ConfigMap named ovn-ic in the global cluster with the following command.

   kubectl delete cm ovn-ic -n cpaas-system

#Configure Cluster Gateway High Availability

To configure the cluster gateway to be highly available after joining the cluster interconnection, you can perform the following steps:

1. Log in to the cluster that needs to be transformed into a High Availability Gateway and execute the following command to change the enable-ic field to false.

   Note: Changing the enable-ic field to false will disrupt the cluster interconnect until it is set to true again.

   kubectl edit cm ovn-ic-config -n kube-system

2. Modify the gateway node configuration by updating the gw-nodes field and separating the gateway nodes with English commas; also change the enable-ic field to true.

   kubectl edit cm ovn-ic-config -n kube-system
   
   # Configuration example
   apiVersion: v1
   data:
     auto-route: "true"
     az-name: docker
     enable-ic: "true"
     gw-nodes: 192.168.188.234,192.168.189.54
     ic-db-host: 192.168.178.97
     ic-nb-port: "6645"
     ic-sb-port: "6646"
   kind: ConfigMap
   metadata:
     creationTimestamp: "2023-06-13T08:01:16Z"
     name: ovn-ic-config
     namespace: kube-system
     resourceVersion: "99671"
     uid: 6163790a-ad9d-4d07-ba82-195b11244983

3. Go to the Pod in cluster ovn-central and execute the ovn-nbctl lrp-get-gateway-chassis {current cluster name}-ts command to verify that the configuration is in effect.

   ovn-nbctl lrp-get-gateway-chassis docker-ts
   
   # Return to the display example. In this case, the values of 100 and 99 are the priority, and the larger the value, the higher the priority of the corresponding gateway node to be used.
   docker-ts-71292a21-131d-492a-9f0c-0611af458950 100
   docker-ts-1de7ee15-f372-4ab9-8c85-e54d61ea18f1 99