Block Storage Disaster Recovery

RBD Mirror is a feature of Ceph Block Storage (RBD) that enables asynchronous data replication between different Ceph clusters, providing cross-cluster Disaster Recovery (DR). Its core function is to synchronize data in a primary-backup mode, ensuring rapid service takeover by the backup cluster when the primary cluster fails.

WARNING

RBD Mirror performs incremental synchronization based on snapshots, with a default snapshot interval of once per hour (configurable). The differential data between primary and backup clusters typically corresponds to writes within one snapshot cycle.
RBD Mirror only provides underlying storage data backup and does not handle Kubernetes resource backups. Please use the platform's Backup and Restore feature to back up PVC and PV resources.

Terminology

Term	Explanation
Primary Cluster	The cluster currently providing storage services.
Secondary Cluster	The standby cluster used for backup purposes.

Backup Configuration

Prerequisites

Prepare two clusters capable of deploying Alauda Build of Rook-Ceph: a Primary cluster and a Secondary cluster, with network connectivity between them.
Both clusters must run the same platform version (v3.12 or later).
Create distributed storage services in both Primary and Secondary clusters.
Create block storage pools with identical names in both Primary and Secondary clusters.
Please ensure that the following three images have been uploaded to the platform's private image repository:
- quay.io/csiaddons/k8s-controller:v0.5.0
- quay.io/csiaddons/k8s-sidecar:v0.8.0
- quay.io/brancz/kube-rbac-proxy:v0.8.0

Procedures

Enable Mirroring for Primary Cluster's Block Storage Pool

Execute the following command on the Primary cluster's Control node:

Command

Output

kubectl -n rook-ceph patch cephblockpool <block-pool-name> \
--type merge -p '{"spec":{"mirroring":{"enabled":true,"mode":"image"}}}'

Parameters:

<block-pool-name>: Block storage pool name.

Retrieve Peer Token

This token serves as the critical credential for establishing mirror connections between clusters.

Execute the following command on the Primary cluster's Control node:

Command

Output

kubectl get secret -n rook-ceph \
$(kubectl get cephblockpool.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.info.rbdMirrorBootstrapPeerSecretName}') \
-o jsonpath='{.data.token}' | base64 -d

Parameters:

<block-pool-name>: Block storage pool name.

Create Peer Token Secret in Secondary Cluster

Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl -n rook-ceph create secret generic rbd-primary-site-secret \
--from-literal=token=<token> \
--from-literal=pool=<block-pool-name>

Parameters:

<token>: Token obtained from Step 2.
<block-pool-name>: Block storage pool name.

Enable Mirroring for Secondary Cluster's Block Storage Pool

在 Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl -n rook-ceph patch cephblockpool <block-pool-name> --type merge -p \
'{
  "spec": {
    "mirroring": {
      "enabled": true,
      "mode": "image",
      "peers": {
        "secretNames": [
          "rbd-primary-site-secret"
        ]
      }
    }
  }
}'

Parameters:

<block-pool-name>: Block storage pool name.

Deploy Mirror Daemon in Secondary Cluster

This daemon is responsible for monitoring and managing RBD mirror synchronization processes, including data synchronization and error handling.

Execute the following command on the Secondary cluster's Control node:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephRBDMirror
metadata:
  name: rbd-mirror
  namespace: rook-ceph
spec:
  count: 1
EOF

Verify Mirror Status

Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl get cephblockpools.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.mirroringStatus.summary}'

Parameters:

<block-pool-name>: Block storage pool name.

Enable Replication Sidecar

This feature enables efficient data replication and synchronization without interrupting primary application operations, enhancing system reliability and availability.

Deploy csiaddons-controller

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Click to view

kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/crds.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/rbac.yaml

cat << EOF | kubectl apply -f -
apiVersion: v1
data:
  controller_manager_config.yaml: |
    apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
    kind: ControllerManagerConfig
    health:
      healthProbeBindAddress: :8081
    metrics:
      bindAddress: 127.0.0.1:8080
    webhook:
      port: 9443
    leaderElection:
      leaderElect: true
      resourceName: e8cd140a.openshift.io
kind: ConfigMap
metadata:
  name: csi-addons-manager-config
  namespace: csi-addons-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: csi-addons
  name: csi-addons-controller-manager
  namespace: csi-addons-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: csi-addons
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: manager
      labels:
        app.kubernetes.io/name: csi-addons
    spec:
      containers:
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        image: <registry>/brancz/kube-rbac-proxy:v0.8.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=127.0.0.1:8080
        - --leader-elect
        command:
        - /manager
        image: <registry>/csiaddons/k8s-controller:v0.5.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      serviceAccountName: csi-addons-controller-manager
      terminationGracePeriodSeconds: 10
EOF

Parameters:

<registry>: Registry address of platform.

Enable csi sidecar

Execute the following commands on both Primary and Secondary clusters' Control nodes:

kubectl patch cm rook-ceph-operator-config -n rook-ceph --type json --patch \
'[
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_OMAP_GENERATOR",
    "value": "true"
  },
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_CSIADDONS",
    "value": "true"
  }
]'

Create VolumeReplicationClass

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
  name: rbd-volumereplicationclass
spec:
  provisioner: rook-ceph.rbd.csi.ceph.com
  parameters:
    mirroringMode: snapshot
    schedulingInterval: "<scheduling-interval>"
    replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
    replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF

<scheduling-interval>: Scheduling interval, (e.g., schedulingInterval: "1h" indicates execution every 1 hour.)

Enable Mirror for PVC

Execute the following command on the Primary cluster's Control node:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF

<vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
<namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
<pvc-name>: The name of the PVC for which Mirror needs to be enabled.

Note After enabling, the RBD image in the Secondary cluster becomes read-only.

Failover

When the Primary cluster fails, it is necessary to switch the primary-backup relationship of the RBD image.

Prerequisites

The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

创建 VolumeReplication

Execute the following command on the Secondary cluster's Control node:

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <mirror-pvc-name>
  replicationHandle: ""
  replicationState: primary
  volumeReplicationClass: rbd-volumereplicationclass
EOF

<vr-name>: VolumeReplication name.
<namespace>: PVC namespace.
<mirror-pvc-name>: The name of the PVC.

Note After creation, the RBD image on the Secondary cluster becomes primary and is writable.

Block Storage Disaster Recovery

WARNING

RBD Mirror performs incremental synchronization based on snapshots, with a default snapshot interval of once per hour (configurable). The differential data between primary and backup clusters typically corresponds to writes within one snapshot cycle.
RBD Mirror only provides underlying storage data backup and does not handle Kubernetes resource backups. Please use the platform's Backup and Restore feature to back up PVC and PV resources.

Terminology

Term	Explanation
Primary Cluster	The cluster currently providing storage services.
Secondary Cluster	The standby cluster used for backup purposes.

Backup Configuration

Prerequisites

Prepare two clusters capable of deploying Alauda Build of Rook-Ceph: a Primary cluster and a Secondary cluster, with network connectivity between them.
Both clusters must run the same platform version (v3.12 or later).
Create distributed storage services in both Primary and Secondary clusters.
Create block storage pools with identical names in both Primary and Secondary clusters.
Please ensure that the following three images have been uploaded to the platform's private image repository:
- quay.io/csiaddons/k8s-controller:v0.5.0
- quay.io/csiaddons/k8s-sidecar:v0.8.0
- quay.io/brancz/kube-rbac-proxy:v0.8.0

Procedures

Enable Mirroring for Primary Cluster's Block Storage Pool

Execute the following command on the Primary cluster's Control node:

Command

Output

kubectl -n rook-ceph patch cephblockpool <block-pool-name> \
--type merge -p '{"spec":{"mirroring":{"enabled":true,"mode":"image"}}}'

Parameters:

<block-pool-name>: Block storage pool name.

Retrieve Peer Token

This token serves as the critical credential for establishing mirror connections between clusters.

Execute the following command on the Primary cluster's Control node:

Command

Output

kubectl get secret -n rook-ceph \
$(kubectl get cephblockpool.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.info.rbdMirrorBootstrapPeerSecretName}') \
-o jsonpath='{.data.token}' | base64 -d

Parameters:

<block-pool-name>: Block storage pool name.

Create Peer Token Secret in Secondary Cluster

Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl -n rook-ceph create secret generic rbd-primary-site-secret \
--from-literal=token=<token> \
--from-literal=pool=<block-pool-name>

Parameters:

<token>: Token obtained from Step 2.
<block-pool-name>: Block storage pool name.

Enable Mirroring for Secondary Cluster's Block Storage Pool

在 Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl -n rook-ceph patch cephblockpool <block-pool-name> --type merge -p \
'{
  "spec": {
    "mirroring": {
      "enabled": true,
      "mode": "image",
      "peers": {
        "secretNames": [
          "rbd-primary-site-secret"
        ]
      }
    }
  }
}'

Parameters:

<block-pool-name>: Block storage pool name.

Deploy Mirror Daemon in Secondary Cluster

This daemon is responsible for monitoring and managing RBD mirror synchronization processes, including data synchronization and error handling.

Execute the following command on the Secondary cluster's Control node:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephRBDMirror
metadata:
  name: rbd-mirror
  namespace: rook-ceph
spec:
  count: 1
EOF

Verify Mirror Status

Execute the following command on the Secondary cluster's Control node:

Command

Output

kubectl get cephblockpools.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.mirroringStatus.summary}'

Parameters:

<block-pool-name>: Block storage pool name.

Enable Replication Sidecar

This feature enables efficient data replication and synchronization without interrupting primary application operations, enhancing system reliability and availability.

Deploy csiaddons-controller

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Click to view

kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/crds.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/rbac.yaml

cat << EOF | kubectl apply -f -
apiVersion: v1
data:
  controller_manager_config.yaml: |
    apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
    kind: ControllerManagerConfig
    health:
      healthProbeBindAddress: :8081
    metrics:
      bindAddress: 127.0.0.1:8080
    webhook:
      port: 9443
    leaderElection:
      leaderElect: true
      resourceName: e8cd140a.openshift.io
kind: ConfigMap
metadata:
  name: csi-addons-manager-config
  namespace: csi-addons-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: csi-addons
  name: csi-addons-controller-manager
  namespace: csi-addons-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: csi-addons
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: manager
      labels:
        app.kubernetes.io/name: csi-addons
    spec:
      containers:
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        image: <registry>/brancz/kube-rbac-proxy:v0.8.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=127.0.0.1:8080
        - --leader-elect
        command:
        - /manager
        image: <registry>/csiaddons/k8s-controller:v0.5.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      serviceAccountName: csi-addons-controller-manager
      terminationGracePeriodSeconds: 10
EOF

Parameters:

<registry>: Registry address of platform.

Enable csi sidecar

Execute the following commands on both Primary and Secondary clusters' Control nodes:

kubectl patch cm rook-ceph-operator-config -n rook-ceph --type json --patch \
'[
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_OMAP_GENERATOR",
    "value": "true"
  },
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_CSIADDONS",
    "value": "true"
  }
]'

Create VolumeReplicationClass

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
  name: rbd-volumereplicationclass
spec:
  provisioner: rook-ceph.rbd.csi.ceph.com
  parameters:
    mirroringMode: snapshot
    schedulingInterval: "<scheduling-interval>"
    replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
    replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF

<scheduling-interval>: Scheduling interval, (e.g., schedulingInterval: "1h" indicates execution every 1 hour.)

Enable Mirror for PVC

Execute the following command on the Primary cluster's Control node:

Command

Output

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF

<vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
<namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
<pvc-name>: The name of the PVC for which Mirror needs to be enabled.

Note After enabling, the RBD image in the Secondary cluster becomes read-only.

Failover

When the Primary cluster fails, it is necessary to switch the primary-backup relationship of the RBD image.

Prerequisites

The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

创建 VolumeReplication

Execute the following command on the Secondary cluster's Control node:

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <mirror-pvc-name>
  replicationHandle: ""
  replicationState: primary
  volumeReplicationClass: rbd-volumereplicationclass
EOF

<vr-name>: VolumeReplication name.
<namespace>: PVC namespace.
<mirror-pvc-name>: The name of the PVC.

Note After creation, the RBD image on the Secondary cluster becomes primary and is writable.

ACP CLI (ac)

Node Management

Managed Clusters

Import Clusters

Public Cloud Cluster Initialization

Network Initialization

Storage Initialization

How to

How to

Backup Management

Recovery Management

Guides

How To

Kube OVN

alb

Trouble Shooting

Concepts

Guides

How To

Troubleshooting

Object Storage

Guides

How To

Install

Concepts

Guides

How To

Disaster Recovery

Concepts

Guides

How To

Guides

How To

ALB Operator

Compliance

HowTo

API Refiner

User

Guides

Group

Guides

Role

Guides

IDP

Guides

Troubleshooting

User Policy

Guides

Overview

Images

Guides

How To

Virtual Machine

Guides

How To

Troubleshooting

Network

Guides

How To

Storage

Guides

Backup and Recovery

Guides

Concepts

Namespaces

Creating Applications

Operation and Maintaining Applications

Application Rollout

KEDA(Kubernetes Event-driven Autoscaling)

How To

Workloads

Configurations

Application Observability

How To

How To

Install

How To

Overview

Install

Upgrade