This page describes how torecover orrestore Cassandra in multiple regions.

In a multi-region deployment, Apigee hybrid is deployed in multiple geographic locations across different datacenters. If one or more regions fail, but healthy regions remain, you can use a healthy region torecover failed Cassandra regions with the latest data.

In the event of a catastrophic failure of all hybrid regions, Cassandra can berestored. It is important to note that, if you have multiple Apigee organizations in your deployment, the restore process restores data forall the organizations. In a multi-organization setup, restoring only a specific organization isnot supported.

This topic describes both approaches to salvaging failed region(s):

• Recover failed region(s) - Describes the steps to recover failed region(s) based on a healthy region.
• Restore failed region(s) - Describes the steps to restore failed region(s) from a backup. This approach is only required ifall hybrid regions are impacted.

Recover failed region(s)

To recover failed region(s) from a healthy region, perform the following steps:

1. Redirect the API traffic from the impacted region(s) to the good working region. Plan the capacity accordingly to support the diverted traffic from failed region(s).
2. Decommission the impacted region. For each impacted region, follow the steps outlined inDecommission a hybrid region. Wait for decommissioning to complete before moving on to the next step.

3. Restore the impacted region. To restore, create a new region, as described inMulti-region deployment on GKE, GKE on-prem, and AKS.

Restoring from a backup

kubectl-nAPIGEE_NAMESPACEgetapigeeds-o=jsonpath='{.items[].spec.components.cassandra.hostNetwork}'

1. Open the overrides file for the region you wish to restore.
2. Setcassandra:hostNetwork tofalse.
3. Apply the overrides file:

   helm upgrade datastore apigee-datastore/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

4. Before continuing, check to make sure thehostNetwork is set tofalse:

   kubectl-nAPIGEE_NAMESPACEgetapigeeds-o=jsonpath='{.items[].spec.components.cassandra.hostNetwork}'

5. Delete hybrid from the region you are restoring:

   helm deleteDATASTORE_RELEASE_NAME \  --namespaceAPIGEE_NAMESPACE

   WhereDATASTORE_RELEASE_NAME is the release name of the datastore you installed Cassandra in the region, for exampledatastore-region1.

   Note: Be careful which release of the datastore component you delete. You can see your releases with the commandhelm -nAPIGEE_NAMESPACE ls.

6. Restore the desired region from a backup. For more information, seeRestoring a region from a backup.

7. Remove the dead nodes from the deleted region(s):
   1. Using thenodetool status command, identify all the nodes withDN status in the deleted region(s):

      kubectlexecapigee-cassandra-default-0-nAPIGEE_NAMESPACE--nodetool-uAPIGEE_JMX_USER-pwAPIGEE_JMX_PASSWORDstatus

      where:

      • APIGEE_JMX_USER is the username for the Cassandra JMX operations user. Used to authenticate and communicate with the Cassandra JMX interface. Seecassandra:auth:jmx:username.
      • APIGEE_JMX_PASSWORD is the password for the Cassandra JMX operations user. Seecassandra:auth:jmx:password.
   2. Remove each node withDN status in the deleted region(s):

      kubectlexecapigee-cassandra-default-0-nAPIGEE_NAMESPACE--nodetool-uAPIGEE_JMX_USER-pwAPIGEE_JMX_PASSWORDremovenodeNODE_HOST_ID

      ReplaceNODE_HOST_ID with the Host ID of each node you identified in the previous step. For example:

      kubectlexecapigee-cassandra-default-0-napigee--nodetool-umy-jmx-id-pwmy-jmx-pwdremovenodeabcdef-f202-438a-8a77-40636317b7c9

   3. Run thenodetool status command again and verify that the nodes withDN status in the deleted region(s) are gone.

      kubectlexecapigee-cassandra-default-0-nAPIGEE_NAMESPACE--nodetool-uAPIGEE_JMX_USER-pwAPIGEE_JMX_PASSWORDstatus

8. Remove the deleted region(s) references and add the restored region(s) references in theKeySpaces metadata.
   1. Get the cassandra datacenter name by using thenodetool status option.

      kubectl exec -nAPIGEE_NAMESPACE -it apigee-cassandra-default-0 -- bash      nodetool  -uAPIGEE_JMX_USER -pwAPIGEE_JMX_PASSWORD status |grep -i Datacenter

      where:

      • APIGEE_JMX_USER is the username for the Cassandra JMX operations user. Used to authenticate and communicate with the Cassandra JMX interface. Seecassandra:auth:jmx:username.
      • APIGEE_JMX_PASSWORD is the password for the Cassandra JMX operations user. Seecassandra:auth:jmx:password.
   2. Update theKeySpaces replication.
      1. Create a client container and connect to the Cassandra cluster through the CQL interface.
      2. Get the list of user keyspaces from CQL interface:

         cqlshCASSANDRA_SEED_HOST -uAPIGEE_DDL_USER -pAPIGEE_DDL_PASSWORD      --ssl -e "select keyspace_name from system_schema.keyspaces;"|grep -v system

         where:

         • CASSANDRA_SEED_HOST is the Cassandra multi-region seed host. For most multi-region installations, use the IP address of a host in your first region. SeeConfigure Apigee hybrid for multi-region andcassandra:externalSeedHost.
         • APIGEE_DDL_USER andAPIGEE_DDL_PASSWORD are the admin username and password for the Cassandra Data Definition Language (DDL) user. The default values are "ddl_user" and "iloveapis123".You can check the DDL username and password by checking the value stored in theapigee-datastore-default-credssecret. You must have admin privileges to check this secret. The following command will return base-64 encoded values for the DDL username and password:

           kubectl get secret apigee-datastore-default-creds -nAPIGEE_NAMESPACE -o yaml

           Seecassandra.auth.ddl.password in the Configuration properties reference andCommand Line Options in the Apache Cassandra cqlsh documentation.

      3. For each keyspace, run the following command from the CQL interface to update the replication settings:

         ALTER KEYSPACEKEYSPACE_NAME WITH replication = {'class': 'NetworkTopologyStrategy', 'DATACENTER_NAME':3};

         where:

         • KEYSPACE_NAME is the name of the keyspace listed in the previous step's output.
         • DATACENTER_NAME is name of the Cassandra datacenter you obtained with thenodetool status option in step 8.