Upgrading Apigee hybrid to version 1.3.6

You are currently viewing version 1.3 of the Apigee hybrid documentation.This version is end of life. You should upgrade to a newer version. For more information, seeSupported versions.
Note:This procedure covers upgrading from Apigee hybrid version 1.2.0 to Apigee hybrid version 1.3.6 only.

If you are upgrading from Apigee hybrid version 1.0 or 1.1, you must first upgrade to hybrid version 1.2 before upgrading to version 1.3.6. See the instructions forUpgrading Apigee hybrid to version 1.2.

Upgrading to version 1.3.6 overview.

Upgrading to Apigee hybrid version 1.3.6 requires downtime during the

To minimize downtime in production hybrid environments, make sure you are running at least two clusters, and update the clusters one at a time.

In addition, upon upgrade each component will perform a rolling restart.

The procedures for upgrading Apigee hybrid are organized in the following sections:

  1. Preparation
    1. Create and update service accounts.
    2. Plan environment groups.
    3. Copy and update overrides file.
  2. Upgrade Istio and cert-manager.
  3. Install hybrid runtime version 1.3.
  4. Clean up.
Note:Do not create new environments during the upgrade process.

Prerequisite

Preparation

  1. (Recommended) Make a backup copy of your version 1.2$APIGEECTL_HOME/ directory. For example:
    tar -czvf $APIGEECTL_HOME/../apigeectl-v1.2-backup.tar.gz $APIGEECTL_HOME
  2. (Recommended) Backup your Cassandra database following the instructions inCassandra backup and recovery
  3. Upgrade your Kubernetes platform as follows. Follow your platform's documentation if you need help:
    PlatformUpgrade to version
    GKE1.15.x
    Anthos1.5
    AKS1.16.x usingAnthos attached clusters
  4. If you are not using Apigee Connect in your hybrid installation, enable Apigee Connect.
    1. Check to see if the Apigee Connect API is enabled:
      gcloud services list | grep apigeeconnectapigeeconnect.googleapis.com         Apigee Connect API
    2. If it is not, enable the API:
      gcloud services enable apigeeconnect.googleapis.com --project$PROJECT_ID

      Where$PROJECT_ID is your Google Cloud project ID.

    3. On the command line, get yourgcloud authentication credentials, as the following example shows:

      TOKEN=$(gcloud auth print-access-token)

      To check that your token was populated, useecho, as the following example shows:

      echo $TOKEN

      This should display your token as an encoded string.

      For more information, seegcloud command-line tool overview.

    4. Check to see if Apigee Connect is enabled for your organization:
      curl -H "Authorization: Bearer$TOKEN" \  "https://apigee.googleapis.com/v1/organizations/$ORG_NAME"

      Where$ORG_NAME is the ID of your organization.

      If the output contains:

            "name" : "features.mart.connect.enabled",      "value" : "true"

      Apigee Connect is enabled.

    5. If Apigee Connect is not enabled, assign the Apigee Connect Agent role to the MART service account:
      gcloud projects add-iam-policy-binding$PROJECT_ID \  --member serviceAccount:apigee-mart@$PROJECT_ID.iam.gserviceaccount.com \  --role roles/apigeeconnect.Agent
      Tip: If you need to get the email address of your MART service account, run the following command:
      gcloud iam service-accounts list | grep mart
    6. Enable Apigee Connect with the following command:
      curl -H "Authorization: Bearer$TOKEN" -X PUT \  -H "Content-Type: application/json" \  -d '{    "name" : "'"$ORG_NAME"'",    "properties" : {      "property" : [ {        "name" : "features.hybrid.enabled",        "value" : "true"      }, {        "name" : "features.mart.connect.enabled",        "value" : "true"      } ]    }  }' \  "https://apigee.googleapis.com/v1/organizations/$ORG_NAME"

      If the output contains the two following properties, Apigee Connect was successfully enabled:

            {        "name": "features.mart.connect.enabled",        "value": "true"      },      {        "name": "features.hybrid.enabled",        "value": "true"      }
  5. Create theapigee-watcher service account. Apigee Watcher is a new service account introduced in v1.3. It whaches the synchronizer for org-level changes and applies those changes to configure the Istio ingress.

    From your main hybrid directory:

    ./tools/create-service-account apigee-watcher ./service-accounts
  6. Assign Apigee Runtime Agent role to the Watcher service account:
    gcloud projects add-iam-policy-binding$PROJECT_ID \  --member serviceAccount:apigee-watcher@$PROJECT_ID.iam.gserviceaccount.com \  --role roles/apigee.runtimeAgent

    WherePROJECT_ID is your Google Cloud project ID. If your service account email addresses differ from this pattern, replace them accordingly.

    Tip: If you need to get the email address of your watcher service account, run the following command:
    gcloud iam service-accounts list | grep watcher

    The output should include a list of all the service accounts and their roles, including:

      ...- members:  - serviceAccount:apigee-watcher@hybrid13rc5.iam.gserviceaccount.com  role: roles/apigee.runtimeAgent  ...
  7. Plan yourenvironment groups for routing. Apigee hybrid 1.3 manages base path routing with environment groups instead ofroutingRules. If you are usingroutingRules in your hybrid configuration, design environment groups to replicate your routing.

    You must create at least one environment group.

    SeeAbout environment groups.

    Click to see an example of using environment groups to replaceroutingRules

    In this example, the followingroutingRules in theoverrides.yaml file route requests to thefoo andbar API proxies to theprod-env-1 environment, and thefoo-test proxy to thetest-env environment:

    virtualhosts:  - name: default    hostAliases:      - "*.acme.com"    sslCertPath: ./certs/keystore.pem    sslKeyPath: ./certs/keystore.key    routingRules:      - paths:        - /foo        - /bar      - env: prod-env-1      - paths:        - foo-test      -env: test-env

    To duplicate that in hybrid version 1.3.6:

    1. Deploy thefoo andbar proxies to theprod-env-1 environment.
    2. Deploy thefoo-test andbar proxy to thetest-env environment.
    3. Create an environment group with a hostalias for the proxies, likeapi.acme.com.Tip: Create an environment group with the same name as the virtualhosts name you used in version 1.2.
    4. Revise thevirtualhosts stanza inoverrides.yaml to
      • Replacevirtualhosts:name with the name of the environment group.
      • Remove thevirtualhosts:routingRules.

      For example:

      virtualhosts:  - name: default # environment group name    sslCertPath: ./certs/keystore.pem    sslKeyPath: ./certs/keystore.key
  8. Update your overrides file:
    1. Make a copy of your overrides file.
    2. Updategcp andk8sCluster stanzas.

      The following configuration properties have been replaced in hybrid version 1.3:

      • gcpRegion replaced withgcp:region
      • gcpProjectID replaced withgcp:projectID
      • gcpProjectIDRuntime replaced withgcp:gcpProjectIDRuntime
      • k8sClusterName replaced withk8s:clusterName
      • k8sClusterRegion replaced withk8s:clusterRegion

      For example, replace the following structure:

      gcpRegion:gcp regiongcpProjectID:gcp project IDgcpProjectIDRuntime:gcp project IDk8sClusterName:namek8sClusterRegion:region

      with:

      gcp: projectID:gcp project ID region:gcp region gcpProjectIDRuntime:gcp project ID # optional. This is only required if you                                       # want logger/metrics data to be sent in                                       # different gcp project.k8sCluster: name:gcp project ID region:gcp region
    3. If you do not have a unique instance identifier already in your overrides file, add one:
      # unique identifier for this installation. 63 chars length limitinstanceID:ID

      WhereID is a unique identifier for this hybrid installation, like "my-hybrid-131-installation" or "acmecorp-hybrid-131."

    4. Add the Watcher (apigee-watcher) service account to the overrides file:
      # Note: the SA should have the "Apigee Runtime Agent" rolewatcher: serviceAccountPath: "service account file"
    5. Add the Metrics (apigee-metrics) service account to the overrides file:
      metrics: serviceAccountPath: "service account file"
    6. Updatevirtualhosts: stanza to replaceroutingRuleswith your environment group.
      1. -name:Replace the name with the name of your environment group. You can have multiple name entries, one for each environment group.
      2. hostAliases:[] Delete this line.
      3. Keep (or add) thesslCertPath: andsslKeyPath: entries.
      4. Delete allroutingRules entries.

      For example:

      virtualhosts:  - name: default    hostAliases:      - "*.acme.com"    sslCertPath: ./certs/keystore.pem    sslKeyPath: ./certs/keystore.key    routingRules:      - paths:        - /foo        - /bar      - env: my-environment

      Becomes:

      virtualhosts:  - name: example-env-group    sslCertPath: ./certs/keystore.pem    sslKeyPath: ./certs/keystore.key
    7. Update themart andconnectAgent: stanzas.Note:If you are not planning to use Apigee Connect in your hybrid 1.3 installation, skip this step and follow thetip in step 5 inApigee Connect after upgrading.
      1. Undermart: remove thehostAlias:,sslCertPath:, andsslKeyPath: entries.
      2. Add anconnectAgent: stanza.
      3. UnderconnectAgent: add aserviceAccountPath: entry and provide the path to the service acount file that has the Apigee Connect Agent role assigned to it (usually the MART service account).

      For example:

      mart:  hostAlias: "mart.apigee-hybrid-docs.net"  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json  sslCertPath: ./certs/fullchain.pem  sslKeyPath: ./certs/privkey.key

      Becomes:

      mart:  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.jsonconnectAgent:  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json

Upgrade Istio and cert-manager

Apigee hybrid version 1.3 requires cert-manager v0.14.2 to manage and verify certificates and the Istio distribution provided with Anthos Service Mesh (ASM) version 1.5.7 (or newer) to create and manage the runtime ingress gateway.

Upgrade Istio 1.4.6 to ASM 1.5.7 (or newer)

Traffic disruption may occur during the upgrade process. To minimize the disruption, ensure that at least two replicas of each component (except Citadel) are running.
  1. To minimize downtime, the Istio deployments and HPAs must have at least two replicas each. Run the following commands to determine the number of replicas:
    kubectl -n istio-system get deployments # list of deploymentskubectl -n istio-system get hpa # list of hpa
  2. Edit each deployment that has only one replica and increase thereplicas: to2 or more:
    kubectl -n istio-system edit deploymentname

    For example:

    spec:  progressDeadlineSeconds: 600  replicas: 2
  3. Edit each HPA that has only one replica and increase theminReplicas: to2 or more:
    kubectl -n istio-system edit hpaname

    For example:

    spec:  maxReplicas: 5  minReplicas: 2
  4. Download and install ASM following the install instructions inDownload and install ASM.
  5. After the installation, run the version command to make sure you have 1.5.x installed correctly:
    ./bin/istioctl versionclient version: 1.5.8-asm.0apigee-mart-ingressgateway version:citadel version: 1.4.6galley version: 1.4.6ingressgateway version: 1.5.8-asm.0pilot version: 1.4.6policy version: 1.4.6sidecar-injector version: 1.4.6telemetry version: 1.4.6pilot version: 1.5.8-asm.0data plane version: 1.4.6 (1 proxies), 1.5.8-asm.0 (2 proxies)

Upgrade cert-manager

Apigee hybrid version 1.3 requires cert-manager version 0.14.2. You can follow the complete instructions on the cert-manager site forUpgrading from v0.13 to v0.14. The steps needed for hybrid are provided below.
  1. Delete the current cert-manager deployment:
    kubectl delete -n cert-manager deployment cert-manager cert-manager-cainjector cert-manager-webhook
  2. Verify your version of Kubernetes:
    kubectl version
  3. Run the following command to install cert-manager from Jetstack:
    kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

Install the hybrid runtime

  1. Store the latest version number in a variable:
    export VERSION=$(curl -s \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/current-version.txt?ignoreCache=1)
  2. Check that the variable was populated with a version number. If you want to use a different version, you can save that in environment variable instead. For example:
    echo $VERSION  1.3.6

  3. Download the release package for your operating system:

    Mac 64 bit:

    curl -LO \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_mac_64.tar.gz

    Linux 64 bit:

    curl -LO \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_linux_64.tar.gz

    Mac 32 bit:

    curl -LO \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_mac_32.tar.gz

    Linux 32 bit:

    curl -LO \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_linux_32.tar.gz
  4. Rename your currentapigeectl/ directory to a backup directory name. For example:
    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.2/

  5. Extract the downloaded gzip file contents into your hybrid base directory. For example:

    tar xvzffilename.tar.gz -Chybrid-base-directory
  6. cd to the base directory.

  7. The tar contents are, by default, expanded into a directory with the version and platform in its name. For example:./apigeectl_1.0.0-f7b96a8_linux_64. Rename that directory toapigeectl:

    mv apigeectl_1.0.0-f7b96a8_linux_64 apigeectl
  8. Delete theapigee-resources-install job fromapigee-system:
    kubectl -n apigee-system delete job apigee-resources-install
  9. Delete the older CRD:
    kubectl delete crd apigeetelemetries.apigee.cloud.google.com
  10. Update thecassandra: stanza in your overrides file with anexternalSeedHost property. This property will help ensure your new hybrid version 1.3.6 installation will use the same Kubernetes cluster as your version 1.2 installation. This is a one-time step needed only for upgrade from hybrid version 1.2 to version 1.3.6 (or newer).
    1. Find one of the IP addresses of the existing Cassandra in the same Kubernetes cluster where you are upgrading the 1.2.0 installation.
      kubectl -nnamespace get pods -o wide

      Wherenamespace is your Apigee hybrid namespace.

      Make note of the IP address of a Cassandra node. For example:

      kubectl -n apigee get pods -o wideNAME                  READY   STATUS    RESTARTS   AGE   IP          NODEapigee-cassandra-0    1/1     Running   0          33d10.68.8.24   gke-example-cluster-rc5-apigee-data-c8bf1234-09kcapigee-cassandra-1    1/1     Running   0          16d   10.68.8.33   gke-example-cluster-rc5-apigee-data-c9221ee7-10kcapigee-cassandra-2    1/1     Running   0          23h   10.68.9.11   gke-example-cluster-rc5-apigee-data-d123e456-11kc
    2. Add the value for theexternalSeedHost property:
      cassandra: externalSeedHost:Cassandra_node_IP

      WhereCassandra_node_IP is the IP of the cassandra node (10.68.8.24 in the preceding example).

      Note: You will remove theexternalSeedHost property in a later step after your new Cassandra is running and verified.
  11. In thenewapigeectl/ directory, runapigeectl init,apigeectl apply, andapigeectl check-ready:
    1. Initialize hybrid 1.3.6:
      apigeectl init -foverrides_1.3.yaml

      Whereoverrides_1.3.yaml is your edited overrides.yaml file.

    2. In hybrid version 1.3, the syntax of the--dry-run flag depends on the version ofkubectl you are running. Check the version ofkubectl:
      gcloud version
    3. Check for errors with a dry run:

      kubectl version 1.17 and older:

      apigeectl apply -foverrides_1.3.yaml --dry-run=true

      kubectl version 1.18 and newer:

      apigeectl apply -foverrides_1.3.yaml --dry-run=client
    4. Apply your overrides. Select and follow the instructions for production environments or demo/experimental environments, depending on your installation.During the upgrade process each component will perform a rolling restart. Therefore, for production environments, it is best to apply the upgrade to one component at a time.

      Production

      For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceding to the next component.

      1. Apply your overrides to upgrade Cassandra:
        apigeectl apply -foverrides_1.3.yaml --datastore
      2. Check completion:
        kubectl -nnamespace get pods

        Wherenamespace is your Apigee hybrid namespace.

        Proceed to the next step only when the pods are ready.

      3. Apply your overrides to upgrade Telemetry components and check completion:
        apigeectl apply -foverrides_1.3.yaml --telemetry
        kubectl -nnamespace get pods
      4. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
        apigeectl apply -foverrides_1.3.yaml --org
        kubectl -nnamespace get pods
      5. Apply your overrides to upgrade your environments. You have two choices:
        • Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
          apigeectl apply -foverrides_1.3.yaml --envenv_name
          kubectl -nnamespace get pods

          Whereenv_name is the name of the environment you are upgrading.

        • Apply your overrides to all environments at once and check completion:
          apigeectl apply -foverrides_1.3.yaml --all-envs
          kubectl -nnamespace get pods

      Demo/Experimental

      In most demo or experimental environments, you can apply the overrides to all components at once. If your demo/experimental environment large and complex or closely mimics a production environment, you may want to use the instructions for upgradingproduction environments

      1. apigeectl apply -foverrides_1.3.yaml
      2. Check the status:
        apigeectl check-ready -foverrides_1.3.yaml

      For more instructions seeGKE Hybrid setup - Step 5: Install hybrid on GKE.

    5. Once you have hybrid 1.3 setup up and running, verify that all of the Cassandra nodes (old and new) are part of the same Cassandra cluster. Run the following command on one of the Cassandra nodes:
      kubectl -nnamespace get podskubectl -nnamespace execold Cassandra pod -- nodetool status

      In the following example output, 10.68.8.24 is from version 1.2.0 and is the node IP you used as theexternalSeedHost. 10.68.7.11 is from version 1.3.6:

      Datacenter: dc-1================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address     Load        Tokens       Owns (effective)  Host ID                               RackUN  10.68.8.24  379.41 KiB  256          50.8%             11bbd43b-af64-464b-a96d-0d6dd0521de1  ra-1UN  10.68.7.11  1.35 MiB    256          49.2%             0b4d9e08-f353-413a-b4a9-7d18a8d07e58  ra-1

      If they are not in the same cluster,check theexternalSeedHost value.

    6. Once all the pods are up and running, removeexternalSeedHost from the your overrides file and runapigeectl apply again with--datastore option:
      apigeectl apply --datastore -f overrides_1.3.6.yaml

    Clean up

    Once you have verified all the pods are up and healthy and the ASM endpoints are valid for the new installation, you can clean up:

    • Hybrid 1.2 resources.
    • The older Cassandra instance
    • Istio 1.4.6 resources.

    Delete Hybrid 1.2.0 resources

    Note: This procedure requires theapigeectl version 1.2 command in your backed up directory. If you need to download another copy ofapigeectl version 1.2, you can use the following command:
    curl -LO \    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/1.2.0/apigeectl_your_system.tar.gz

    Whereyour_system is one oflinux_32,linux_64,mac_32, ormac_64.

    1. Remove the 1.2.0 virtualhost routing details:
      $APIGEECTL_HOME-v1.2/apigeectl delete -s virtualhost -f 1.2.0_overrides.yaml

      Where$APIGEECTL_HOME-v1.2 is the directory where you backed up yourapigeectl version 1.2 directory.

    2. If the endpoint is still working as expected and you have verified all the 1.3.0 components are working run the following command to delete the hybrid 1.2.0 resources:
      $APIGEECTL_HOME-v1.2/apigeectl delete -c "mart,connect-agent,synchronizer,runtime,udca,metrics,logger" \  -f 1.2.0_overrides.yaml

    Decommission the older Cassandra instance

    1. cd into the newly installedapigeectldirectory.
    2. Run thetools/cas_cleanup.sh script.

      This script decommissions the old Cassandra pod from the Cassandra ring, deletes old STS, and deletes PVCs.

      bash cas_cleanup.shApigee namespace

    Delete Istio version 1.4.6 resources

    Note: Deleting the Istio Istio version 1.4.6 resources will also delete the MART ingress as well. MART ingress is replaced with Apigee Connect in Apigee hybrid version 1.3.
    1. Run the following command to delete the most recent Istio v.1.4.6 resources:
      kubectl delete all -n istio-system --selector \  'app in (apigee-mart-istio-ingressgateway, galley, security, istio-nodeagent, istio-mixer, sidecarInjectorWebhook, istio-mixer)'
    2. Run the following commands to delete older jobs from the Istio 1.4.6 installation:
      kubectl -n istio-system delete job istio-init-crd-10-1.4.6kubectl -n istio-system delete job istio-init-crd-11-1.4.6kubectl -n istio-system delete job istio-init-crd-14-1.4.6

    Congratulations! You have successfully upgraded to Apigee hybrid version 1.3.6.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-18 UTC.