Multi-region deployment on AKS

You are currently viewing version 1.2 of the Apigee hybrid documentation.This version is end of life. You should upgrade to a newer version. For more information, seeSupported versions.

This topic explains how to set up a multi-region deployment for Apigee hybrid on Microsoft® Azure Kubernetes Service (AKS).

Topologies for multi-region deployment include the following:

  • Active-Active: When you have applications deployed in multiple geographic locations and you require low latency API response for your deployments. You have the option to deploy hybrid in multiple geographic locations nearest to your clients. For example: US West Coast, US East Coast, Europe, APAC.
  • Active-Passive: When you have a primary region and a failover or disaster recovery region.

The regions in a multi-region hybrid deployment communicate via Cassandra, as the following imageshows:

Prerequisites

Before configuring hybrid for multiple regions, you must complete the following prerequisites:

  • Follow thehybrid installation guide for any prerequisites like GCP and org configuration before moving to cluster setup steps.

For detailed information, seeKubernetes documentation.

NOTE: Apigee recommends that you ensure that your servers' times are synchronized.

Several features such as expiration and token revocation rely on accurate system times. If you host the runtime components in different datacenters, then be sure that the system times synchronized.

You can use a tool such asntpdate to verify that server times are synchronized.

Create a virtual network in each region

Create a virtual network for the multi-region deployment. For example, the following example commands create networks in the Central US and Eastern US regions.

NOTE:The steps in this section are provided as examples only. Adapt themas needed to suit your requirements.

Execute this command to create a virtual network in the Eastern US region, with the namemy-hybrid-rg-vnet:

az network vnet create \ --name my-hybrid-rg-vnet \ --location eastus \ --resource-group my-hybrid-rg \ --address-prefixes 120.38.1.0/24 \ --subnet-name my-hybrid-rg-vnet-subnet \ --subnet-prefix 120.38.1.0/26

Execute this command to create a virtual network in the Central US region, with the namemy-hybrid-rg-vnet-ext01:

az network vnet create \ --name my-hybrid-rg-vnet-ext01 \ --location centralus \ --resource-group my-hybrid-rg \ --address-prefixes 192.138.0.0/24 \ --subnet-name my-hybrid-rg-vnet-ext01-subnet \ --subnet-prefix 192.138.0.0/26

Create network peering

Create a network peering between the virtual networks.

NOTE:The steps in this section are provided as examples only. Adapt themas needed to suit your requirements.

Get the virtual network IDs

Peerings are established between virtual network IDs. Get the ID of each virtual network with the az network vnet show command and store the ID in a variable.

Get the ID of the first virtual network, the one namedmy-hybrid-rg-vnet:

vNet1Id=$(az network vnet show \ --resource-group my-hybrid-rg \ --name my-hybrid-rg-vnet \ --query id --out tsv)

Get the ID of the second virtual network, the one namedmy-hybrid-rg-vnet-ext01:

vNet2Id=$(az network vnet show \ --resource-group my-hybrid-rg \ --name my-hybrid-rg-vnet-ext01 \ --query id \ --out tsv)

Create peering from the first to the second virtual network

With the virtual network IDs, you can create a peering from the first virtual netowrk (my-hybrid-rg-vnet) to the second (my-hybrid-rg-vnet-ext01), as shown in the following examples:

If you do not specify the--allow-vnet-access parameter, a peering is established, but no communication can flow through it.
az network vnet peering create \ --name my-hybrid-rg-vnet1-peering \     # The name of the virtual network peering. --resource-group my-hybrid-rg \ --vnet-name my-hybrid-rg-vnet \         # The virtual network name. --remote-vnet $vNet2Id \                # Resource ID of the remote virtual network. --allow-vnet-access

In the command's output, note that thepeeringState isInitiated. The peering remains in the Initiated state until you create the peering from the second virtual network back to the first.

{  ..."peeringState": "Initiated",  ...}

Create a peering from the second virtual network to the first

Example command:

az network vnet peering create \ --name my-hybrid-rg-vnet2-peering \        # The name of the virtual network peering. --resource-group my-hybrid-rg \ --vnet-name my-hybrid-rg-vnet-ext01 \      # The virtual network name. --remote-vnet $vNet1Id \                   # Resource ID of the remote virtual network. --allow-vnet-access

In the command's output, note thatpeeringState isConnected. Azure also changes the peering state of the first to second virtual network peering toConnected.

{  ..."peeringState": "Connected",  ...}

You can also confirm that the peering state for themy-hybrid-rg-vnet1-peering tomy-hybrid-rg-vnet2-peering: peering changed to Connected with the following command:

az network vnet peering show \ --name my-hybrid-rg-vnet1-peering \ --resource-group my-hybrid-rg \ --vnet-name my-hybrid-rg-vnet \ --query peeringState

Expected output:

Connected

Create multi-regional clusters

Set up Kubernetes clusters in multiple regions with differentCIDR blocks. See also theAKS quickstart. Use thelocations and virtual network names you created previously.

Open Cassandra ports 7000 and 7001 between Kubernetes clusters across all regions (7000 may be used as a backup option during troubleshooting)

Configure the multi-region seed host

This section describes how to expand the existing Cassandra cluster to a new region. This setup allows the new region to bootstrap the cluster and join the existing data center. Without this configuration, the multi-region Kubernetes clusters would not know about each other.

  1. Set the kubectl context to the original cluster before retrieving the seed name:
    kubectl config use-contextoriginal-cluster-name

  2. Run the followingkubectl command to identify a seed host address for Cassandra in the current region.

    Aseed host address allows a new regional instance to find the original cluster on the very first startup to learn the topology of the cluster. The seed host address is designated as the contact point in the cluster.

    kubectl get pods -o wide -n apigee | grep apigee-cassandraapigee-cassandra-0  1/1   Running   0   4d17h   120.38.1.9  aks-agentpool-21207753-vmss000000
  3. Decide which of the IPs returned from the previous command will be the multi-region seed host. In this example, where only a single node cassandra cluster is running, the seed host is120.38.1.9.
  4. In data center 2, copy your overrides file to a new file whose name includes the cluster name. For example,overrides_your_cluster_name.yaml.
  5. In data center 2, configurecassandra.multiRegionSeedHost andcassandra.datacenter inoverrides_your_cluster_name.yaml, wheremultiRegionSeedHost is one of the IPs returned by the previous command:
    cassandra:  multiRegionSeedHost:seed_host_IP  datacenter:data_center_name  rack:rack_name

    For example:

    cassandra:  multiRegionSeedHost: 120.38.1.9  datacenter: "centralus"  rack: "ra-1"
  6. In the new data center/region, before you install hybrid, set the same TLS certificates and credentials inoverrides_your_cluster_name.yaml as you set in the first region.NOTE: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in overrides file in the second data center. For details seeConfiguring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

  1. Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
    1. Set the context to the original namespace:
      kubectl config use-contextoriginal-cluster-name
    2. Export the current namespace configuration to a file:
      $ kubectl get namespace -o yaml > apigee-namespace.yaml
    3. Export theapigee-ca secret to a file:
      kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
    4. Set the context to the new region's cluster name:
      kubectl config use-contextnew-cluster-name
    5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
      kubectl apply -f apigee-namespace.yaml

    6. Import the secret to the new cluster:

      kubectl -n cert-manager apply -f apigee-ca.yaml
  2. Install hybrid in the new region. Be sure that theoverrides_your_cluster_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.

    Execute the following two commands to install hybrid in the new region:

    apigeectl init -f overrides_your_cluster_name.yaml
    apigeectl apply -f overrides_your_cluster_name.yaml

  3. Expand all apigee keyspaces.

    The following steps expand the Cassandra data to the new data center:

    1. Open a shell in the Cassandra pod:
      kubectl run -i --tty --restart=Never --rm --image google/apigee-hybrid-cassandra-client:1.0.0 cqlsh
      NOTE: If you don't see a command prompt, try pressing the ENTER key.
    2. Connect to the Cassandra server:
      cqlsh apigee-cassandra-0.apigee-cassandra.apigee.svc.cluster.local -u ddl_user --sslPassword:Connected to apigeecluster at apigee-cassandra-0.apigee-cassandra.apigee.svc.cluster.local:9042.[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.
    3. Get the available keyspaces:
      SELECT * from system_schema.keyspaces ; keyspace_name              | durable_writes | replication----------------------------+----------------+--------------------------------------------------------------------------------------------------------                system_auth |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}              system_schema |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'} cache_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}   kms_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}   kvm_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}         system_distributed |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}                     system |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}                     perses |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'} quota_hybrid_test_7_hybrid |           True |                  {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3'}              system_traces |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}(10 rows)
    4. Update/expand the apigee keyspaces:
      ALTER KEYSPACE cache_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};ALTER KEYSPACE kms_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};ALTER KEYSPACE kvm_hybrid_test_7_hybrid WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};ALTER KEYSPACE perses WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};ALTER KEYSPACE quota_hybrid_test_7_hybrid  WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1':3, 'dc-2':3};
    5. Validate the keyspace expansion:
      SELECT * from system_schema.keyspaces ; keyspace_name              | durable_writes | replication----------------------------+----------------+--------------------------------------------------------------------------------------------------------                system_auth |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}              system_schema |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'} cache_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}   kms_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}   kvm_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}         system_distributed |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}                     system |           True |                                                {'class': 'org.apache.cassandra.locator.LocalStrategy'}                     perses |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'} quota_hybrid_test_7_hybrid |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '3', 'dc-2': '3'}              system_traces |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc-1': '1', 'dc-2': '1'}(10 rows)ddl@cqlsh>
  4. Runnodetool rebuild sequentially on all the nodes in new data center. This may take a few minutes to a few hours depending on the data size.
    kubectl exec apigee-cassandra-0 -n apigee  -- nodetool rebuild -- dc-1
  5. Verify the rebuild processes from the logs. Also, verify the data size using thenodetool status command:
    kubectl logs apigee-cassandra-0 -f -n apigee

    The following example shows example log entries:

    INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for RebuildINFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is completeINFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is completeINFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is completeINFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed
  6. Update the seed hosts. RemovemultiRegionSeedHost: 10.0.0.11 fromoverrides-DC_name.yaml and reapply.Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters inoverrides.yaml and then reapply the configuration.
    apigeectl apply -f overrides-DC_name.yaml

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-18 UTC.