Upgrading Apigee hybrid to version 1.11 Stay organized with collections Save and categorize content based on your preferences.
This procedure covers upgrading from Apigee hybrid version 1.10.x to Apigee hybrid version 1.11.2 and from previous releases of hybrid 1.11.x to version 1.11.2.
Use the same procedures for minor version upgrades (for example version 1.10 to 1.11) and for patch release upgrades (for example 1.11.0 to 1.11.2).
If you are upgrading from Apigee hybrid version 1.9 or older, you must first upgrade to hybrid version 1.10 before upgrading to version 1.11.2. See the instructions forUpgrading Apigee hybrid to version 1.10.
Upgrading to version 1.11.2 overview
Upgrading to Apigee hybrid version 1.11 may require downtime.When upgrading the Apigee controller to version 1.11.2, all Apigee deployments undergo a rolling restart. To minimize downtime in production hybrid environments during a rolling restart, make sure you are running at least two clusters (in the same or different region/data center). Divert all production traffic to a single cluster and take the cluster you are about to upgrade offline, and then proceed with the upgrade process. Repeat the process for each cluster.
Apigee recommends upgrading all clusters as soon as possible to reduce the chances of production impact. There is no time limit on when all remaining clusters must be upgraded after the first one is upgraded. However, until all remaining clusters are upgraded the following operations will be impacted:
- Cassandra backup and restore cannot work with mixed versions. For example, a backup from Hybrid 1.10 cannot be used to restore a Hybrid 1.11 instance.
- Cassandra data streaming will not work between mixed Hybrid versions. Therefore, your Cassandra clusters cannot scale horizontally.
- Region expansion and decommissioning will be impacted, because these operations depend on Cassandra data streaming.
Stricter class instantiation checks in Hyrid 1.11.2-Hotfix.3
Starting with Apigee hybrid 1.11.2-Hotfix.3, theJavaCallout policy now includes additional security during Java class instantiation. The enhanced security measure prevents the deployment of policies that directly or indirectly attempt actions that require permissions that are not allowed.
In the most cases, existing policies will continue to function as expected without any issues. However, there is a possibility that policies relying on third-party libraries, or those with custom code that indirectly triggers operations requiring elevated permissions, could be affected.
Important: To test your installation, follow the procedure inValidate policies after installing 1.11.2-Hotfix.3 to validate policy behavior.The procedures for upgrading Apigee hybrid are organized in the following sections:
Prerequisites
These upgrade instructions assume you have Apigee hybrid version 1.10.x installed and wish to upgrade it to version 1.11.2. If you are updating from an earlier version, see the instructions forUpgrading Apigee hybrid to version 1.10.
Helm charts andapigeectl
In version 1.11 you can choose to install and manage Apigee hybrid with either Helm charts orapigeectl. Apigee recommends using Helm to manage your installation.
Migrating to v1.11 with Helm from v1.10 withapigeectl
To upgrade to Apigee hybrid v1.11 managed by Helm from a hybrid v1.10 installation managed withapigeectl:- First, migrate your v1.10 installation Helm by following the instructions inMigrate Apigee hybrid to Helm charts from
apigeectl. - Follow the instructions for Helm charts below to upgrade your installation.Note: When upgrading a migrated installation, you will need to insert a
--forceflag in thehelm upgradecommands.
Prepare to upgrade to version 1.11
Note: When migrating from Apigee hybrid v1.10 to v1.11, after transitioning fromapigeectl to Helm charts in the previous step, you must follow the instructions under the helm tab. After transitioning to Helm you must continue with Helm in further upgrades.Back up your hybrid installation (recommended)
Helm
- These instructions use the environment variableAPIGEE_HELM_CHARTS_HOME for the directory in your file system where you have installed the Helm charts. If needed, change directory into this directory and define the variable with the following command:
Linux
export APIGEE_HELM_CHARTS_HOME=$PWD
echo$APIGEE_HELM_CHARTS_HOMEMac OS
export APIGEE_HELM_CHARTS_HOME=$PWD
echo$APIGEE_HELM_CHARTS_HOMEWindows
set APIGEE_HELM_CHARTS_HOME=%CD%
echo %APIGEE_HELM_CHARTS_HOME% - Make a backup copy of your version 1.10
$APIGEE_HELM_CHARTS_HOME/directory. You can use any backup process. For example, you can create atarfile of your entire directory with:tar -czvf$APIGEE_HELM_CHARTS_HOME/../apigee-helm-charts-v1.10-backup.tar.gz$APIGEE_HELM_CHARTS_HOME - Back up your Cassandra database following the instructions inCassandra backup and recovery.
- If you are using service cert files (
.json) in your overrides to authenticate service accounts, make sure your service account cert files reside in the correct Helm chart directory. Helm charts cannot read files outside of each chart directory.This step is not needed if you are using Kubernetes secrets or Workload Identity to authenticate service accounts.
The following table shows the destination for each service account file, depending on your type of installation:
Prod
Service account Default filename Helm chart directory apigee-cassandraPROJECT_ID-apigee-cassandra.json$APIGEE_HELM_CHARTS_HOME/apigee-datastore/apigee-loggerPROJECT_ID-apigee-logger.json$APIGEE_HELM_CHARTS_HOME/apigee-telemetry/apigee-martPROJECT_ID-apigee-mart.json$APIGEE_HELM_CHARTS_HOME/apigee-org/apigee-metricsPROJECT_ID-apigee-metrics.json$APIGEE_HELM_CHARTS_HOME/apigee-telemetry/apigee-runtimePROJECT_ID-apigee-runtime.json$APIGEE_HELM_CHARTS_HOME/apigee-envapigee-synchronizerPROJECT_ID-apigee-synchronizer.json$APIGEE_HELM_CHARTS_HOME/apigee-env/apigee-udcaPROJECT_ID-apigee-udca.json$APIGEE_HELM_CHARTS_HOME/apigee-org/apigee-watcherPROJECT_ID-apigee-watcher.json$APIGEE_HELM_CHARTS_HOME/apigee-org/Non-prod
Make a copy of the
apigee-non-prodservice account file in each of the following directories:Service account Default filename Helm chart directories apigee-non-prodPROJECT_ID-apigee-non-prod.json$APIGEE_HELM_CHARTS_HOME/apigee-datastore/$APIGEE_HELM_CHARTS_HOME/apigee-telemetry/$APIGEE_HELM_CHARTS_HOME/apigee-org/$APIGEE_HELM_CHARTS_HOME/apigee-env/ - Make sure that your TLS certificate and key files (
.crt,.key, and/or.pem) reside in the$APIGEE_HELM_CHARTS_HOME/apigee-virtualhost/directory.
apigeectl
The examples in these instructions use the following directory structure. Your installation may be different. Adjust the instructions to the structure of your installation.
hybrid-v1.11-root-directory/└── apigeectl/ └── config/ └── plugins/ └── templates/ └── tools/└── hybrid-files/ └── overrides/ └── service-accounts/ └── certs/hybrid-v1.10-root-directory/
- These instructions use the environment variableAPIGEECTL_HOME for the directory in your file system where you have installed
apigeectl. If needed, change directory into yourapigeectldirectory and define the variable with the following command:Linux
export APIGEECTL_HOME=$PWD
echo$APIGEECTL_HOMEMac OS
export APIGEECTL_HOME=$PWD
echo$APIGEECTL_HOMEWindows
set APIGEECTL_HOME=%CD%
echo %APIGEECTL_HOME% - Make a backup copy of your version 1.10
$APIGEECTL_HOME/directory. For example:tar -czvf$APIGEECTL_HOME/../apigeectl-v1.10-backup.tar.gz$APIGEECTL_HOME - Back up your Cassandra database following the instructions inCassandra backup and recovery.
Upgrade your Kubernetes version
Check your Kubernetes platform version and if needed, upgrade your Kubernetes platform to a version that is supported by both hybrid 1.10 and hybrid 1.11. Follow your platform's documentation if you need help.
Click to expand a list of supported platforms
| 1.10 | 1.11 | 1.12 | |||
|---|---|---|---|---|---|
| GKE on Google Cloud | 1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| GKE on AWS | 1.13.x (K8s v1.24.x) 1.14.x (K8s v1.25.x) 1.26.x(12) 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.14.x (K8s v1.25.x) 1.26.x(12) 1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.26.x(12) 1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| GKE on Azure | 1.13.x 1.14.x 1.26.x(12) 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.14.x 1.26.x(12) 1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.26.x(12) 1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| Google Distributed Cloud (software only) on VMware(1)(13) | 1.13.x 1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(12) 1.29.x(≥ 1.12.1) | ||
| Google Distributed Cloud (software only) on bare metal(1) | 1.13.x 1.14.x (K8s v1.25.x) 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.27.x(12)(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(12)(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(12) 1.29.x(≥ 1.12.1) | ||
| EKS(7) | 1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| AKS(7) | 1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | 1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | 1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| OpenShift(7) | 4.11 4.12 4.14(≥ 1.10.5) 4.15(≥ 1.10.5) | 4.12 4.13 4.14 4.15(≥ 1.11.2) 4.16(≥ 1.11.2) | 4.12 4.13 4.14 4.15 4.16(≥ 1.12.1) | ||
| Rancher Kubernetes Engine (RKE) | v1.26.2+rke2r1 1.27.x(≥ 1.10.5) 1.28.x(≥ 1.10.5) | v1.26.2+rke2r1 v1.27.x 1.28.x(≥ 1.11.2) 1.29.x(≥ 1.11.2) | v1.26.2+rke2r1 v1.27.x 1.28.x 1.29.x(≥ 1.12.1) | ||
| VMware Tanzu | N/A | N/A | v1.26.x | ||
Components | 1.10 | 1.11 | 1.12 | ||
| Anthos Service Mesh | 1.17.x(10) | 1.18.x(10) | 1.19.x(10) | ||
| JDK | JDK 11 | JDK 11 | JDK 11 | ||
| cert-manager | 1.10.x 1.11.x 1.12.x | 1.11.x 1.12.x 1.13.x | 1.11.x 1.12.x 1.13.x | ||
| Cassandra | 3.11 | 3.11 | 4.0 | ||
| Kubernetes | 1.24.x 1.25.x 1.26.x | 1.25.x 1.26.x 1.27.x | 1.26.x 1.27.x 1.28.x | ||
| kubectl | 1.27.x 1.28.x 1.29.x | ||||
| Helm | 3.10+ | 3.10+ | 3.14.2+ | ||
| Secret Store CSI driver | 1.3.4 | 1.4.1 | |||
| Vault | 1.13.x | 1.15.2 | |||
(1) On Anthos on-premises (Google Distributed Cloud) version 1.13, follow these instructions to avoid conflict with (2) Support available with Apigee hybrid version 1.7.2 and newer. (3) The official EOL dates for Apigee hybrid versions 1.10 and older have been reached. Regular monthly patches are no longer available. These releases are no longer officially supported except for customers with explicit and official exceptions for continued support. Other customers must upgrade. (4)Anthos on-premises (Google Distributed Cloud) versions 1.12 and earlier are out of support. See the Distributed Cloud on bare metalVersion Support Policy and the Distributed Cloud on VMwaresupported versions list. (5)Google Distributed Cloud on bare metal or VMware requires Anthos Service Mesh 1.14 or later. We recommend that you upgrade to hybrid v1.8 and switch to Apigee ingress gateway which no longer requires you to install Anthos Service Mesh on your hybrid cluster. (6) Support available with Apigee hybrid version 1.8.4 and newer. (7) Not supported with Apigee hybrid version 1.8.4 and newer. (8) Support available with Apigee hybrid version 1.7.6 and newer. (9) Not supported with Apigee hybrid version 1.8.5 and newer. (10) Anthos Service Mesh is automatically installed with Apigee hybrid 1.9 and newer. (11) Support available with Apigee hybrid version 1.9.2 and newer. (12) GKE on AWS version numbers now reflect the Kubernetes versions. SeeGKE Enterprise version and upgrade support for version details and recommended patches. (13) Vault is not certified on Google Distributed Cloud for VMware. (14) Support available with Apigee hybrid version 1.10.5 and newer. (15) Support available with Apigee hybrid version 1.11.2 and newer. (16) Support available with Apigee hybrid version 1.12.1 and newer. | |||||
About attached clusters
For Apigee hybrid versions 1.7.x and older, you must useGKE attached clusters if you want to run Apigee hybrid in a multi-cloud context on Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or another supported third-party Kubernetes service provider. Cluster attachment allows Google to measure the usage of Anthos Service Mesh.
For Apigee hybrid version 1.8.x, GKE attached clusters are required if you are using Anthos Service Mesh for your ingress gateway. If you are using Apigee ingress gateway, attached clusters are optional.
Note: Hybrid installations are not currently supported on Autopilot clusters.Install the hybrid 1.11.2 runtime
Caution:Do not create new environments during the upgrade process.Helm
Prepare for the Helm charts upgrade
This upgrade procedure assumes you are using the same namespace and service accounts for the upgraded installation. If you are making any configuration changes, be sure to reflect those changes in your overrides file before installing the Helm charts.- Make the following change to youroverrides.yaml file to enable the
apigee-operatorandapigee-envcharts ot use the correct tag,1.11.2-hotfix.3:ao: image: url: "gcr.io/apigee-release/hybrid/apigee-operators" tag: "1.11.2-hotfix.3"runtime: image: url: "gcr.io/apigee-release/hybrid/apigee-runtime" tag: "1.11.2-hotfix.3"
- Pull the Apigee Helm charts.
Apigee hybrid charts are hosted inGoogle Artifact Registry:
oci://us-docker.pkg.dev/apigee-release/apigee-hybrid-helm-chartsUsing the
pullcommand, copy all of the Apigee hybrid Helm charts to your local storage with the following command:exportCHART_REPO=oci://us-docker.pkg.dev/apigee-release/apigee-hybrid-helm-charts
exportCHART_VERSION=1.11.2helm pull$CHART_REPO/apigee-operator --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-datastore --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-env --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-ingress-manager --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-org --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-redis --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-telemetry --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-virtualhost --version$CHART_VERSION --untar - Install cert-manager if needed.
If you need to upgrade your cert-manager version, install the new version with the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml
- Install the updated Apigee CRDs:Note: From this step onwards, all commands should be run under the chart repo root directory.Note: This is the only supported method for installing Apigee CRDs. Do not use
kubectl applywithout-k, do not omit--server-side.Note: This step requires elevated cluster permissions.Use the
kubectldry-run feature by running the following command:kubectl apply -k apigee-operator/etc/crds/default/ --server-side --force-conflicts --validate=false --dry-run
After validating with the dry-run command, run the following command:
kubectl apply -k apigee-operator/etc/crds/default/ --server-side --force-conflicts --validate=false
- Validate the installation with the
kubectl get crdscommand:kubectl get crds | grep apigee
Your output should look something like the following:
apigeedatastores.apigee.cloud.google.com 2023-10-09T14:48:30Zapigeedeployments.apigee.cloud.google.com 2023-10-09T14:48:30Zapigeeenvironments.apigee.cloud.google.com 2023-10-09T14:48:31Zapigeeissues.apigee.cloud.google.com 2023-10-09T14:48:31Zapigeeorganizations.apigee.cloud.google.com 2023-10-09T14:48:32Zapigeeredis.apigee.cloud.google.com 2023-10-09T14:48:33Zapigeerouteconfigs.apigee.cloud.google.com 2023-10-09T14:48:33Zapigeeroutes.apigee.cloud.google.com 2023-10-09T14:48:33Zapigeetelemetries.apigee.cloud.google.com 2023-10-09T14:48:34Zcassandradatareplications.apigee.cloud.google.com 2023-10-09T14:48:35Z
Check the labels on the cluster nodes. By default, Apigee schedules data pods on nodes with the label
cloud.google.com/gke-nodepool=apigee-dataand runtime pods are scheduled on nodes with the labelcloud.google.com/gke-nodepool=apigee-runtime. You can customize your node pool labels in theoverrides.yamlfile.For more information, see Configuring dedicated node pools.
Install the Apigee hybrid Helm charts
Note: Before executing any of the Helm upgrade/install commands,use the Helm dry-run feature by adding--dry-run at the end ofthe command. Seehelm -h to list supported commands, options,and usage.- If you have not, navigate into your
APIGEE_HELM_CHARTS_HOMEdirectory. Run the following commands from that directory. - Upgrade the Apigee Operator/Controller:Note: This step requires elevated cluster permissions. Run
helm -horhelm upgrade -hfor detailsDry run:
helm upgrade operator apigee-operator/ \ --install \ --create-namespace \ --namespace apigee-system \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgrade operator apigee-operator/ \ --install \ --create-namespace \ --namespace apigee-system \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgrade operator apigee-operator/ \ --install \ --create-namespace \ --namespace apigee-system \ --force \ -fOVERRIDES_FILE
Verify Apigee Operator installation:
helm ls -n apigee-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIONoperator apigee-system 3 2023-06-26 00:42:44.492009 -0800 PST deployed apigee-operator-1.11.2 1.11.2
Verify it is up and running by checking its availability:
kubectl -n apigee-system get deploy apigee-controller-manager
NAME READY UP-TO-DATE AVAILABLE AGEapigee-controller-manager 1/1 1 1 7d20h
- Upgrade the Apigee datastore:
Dry run:
helm upgrade datastore apigee-datastore/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgrade datastore apigee-datastore/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgrade datastore apigee-datastore/ \ --install \ --namespaceapigee \ --force \ -fOVERRIDES_FILE
Verify
apigeedatastoreis up and running by checking its state:kubectl -n apigee get apigeedatastore default
NAME STATE AGEdefault running 2d
- Upgrade Apigee telemetry:
Dry run:
helm upgrade telemetry apigee-telemetry/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgrade telemetry apigee-telemetry/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgrade telemetry apigee-telemetry/ \ --install \ --namespaceapigee \ --force \ -fOVERRIDES_FILE
Verify it is up and running by checking its state:
kubectl -n apigee get apigeetelemetry apigee-telemetry
NAME STATE AGEapigee-telemetry running 2d
- Upgrade Apigee Redis:
Dry run:
helm upgrade redis apigee-redis/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgrade redis apigee-redis/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgrade redis apigee-redis/ \ --install \ --namespaceapigee \ --force \ -fOVERRIDES_FILE
Verify it is up and running by checking its state:
kubectl -n apigee get apigeeredis default
NAME STATE AGEdefault running 2d
- Upgrade Apigee ingress manager:
Dry run:
helm upgrade ingress-manager apigee-ingress-manager/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgrade ingress-manager apigee-ingress-manager/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgrade ingress-manager apigee-ingress-manager/ \ --install \ --namespaceapigee \ --force \ -fOVERRIDES_FILE
Verify it is up and running by checking its availability:
kubectl -n apigee get deployment apigee-ingressgateway-manager
NAME READY UP-TO-DATE AVAILABLE AGEapigee-ingressgateway-manager 2/2 2 2 2d
- Upgrade the Apigee organization:
Dry run:
helm upgradeORG_NAME apigee-org/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE \ --dry-run
Upgrade the chart:
Note: For installations migrated fromhelm upgradeORG_NAME apigee-org/ \ --install \ --namespaceapigee \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgradeORG_NAME apigee-org/ \ --install \ --namespaceapigee \ --force \ -fOVERRIDES_FILE
Verify it is up and running by checking the state of the respective org:
kubectl -n apigee get apigeeorg
NAME STATE AGEapigee-org1-xxxxx running 2d
- Upgrade the environment.
You must install one environment at a time. Specify the environment with
--set env=ENV_NAME:Dry run:
helm upgradeENV_RELEASE_NAME apigee-env/ \ --install \ --namespaceapigee \ --set env=ENV_NAME \ -fOVERRIDES_FILE \ --dry-run
- ENV_RELEASE_NAME is the name with which you previously installed the
apigee-envchart. In hybrid v1.10, it is usuallyapigee-env-ENV_NAME. In Hybrid v1.11 and newer it is usuallyENV_NAME. - ENV_NAME is the name of the environment you are upgrading.
- OVERRIDES_FILE is your new overrides file for v.1.11.2
Upgrade the chart:
Note: For installations migrated fromhelm upgradeENV_RELEASE_NAME apigee-env/ \ --install \ --namespaceapigee \ --set env=ENV_NAME \ -fOVERRIDES_FILE
apigeectlto Helm, use:helm upgradeENV_RELEASE_NAME apigee-env/ \ --install \ --namespaceapigee \ --set env=ENV_NAME \ --force \ -fOVERRIDES_FILE
Verify it is up and running by checking the state of the respective env:
kubectl -n apigee get apigeeenv
NAME STATE AGE GATEWAYTYPEapigee-org1-dev-xxx running 2d
- ENV_RELEASE_NAME is the name with which you previously installed the
- Upgrade the environment groups (
virtualhosts).- You must upgrade one environment group (virtualhost) at a time. Specify the environment group with
--set envgroup=ENV_GROUP_NAME. Repeat the following commands for each env group mentioned in the overrides.yaml file:Dry run:
helm upgradeENV_GROUP_RELEASE_NAME apigee-virtualhost/ \ --install \ --namespaceapigee \ --set envgroup=ENV_GROUP_NAME \ -fOVERRIDES_FILE \ --dry-run
ENV_GROUP_RELEASE_NAME is the name with which you previously installed the
apigee-virtualhostchart. In hybrid v1.10, it is usuallyapigee-virtualhost-ENV_GROUP_NAME. In Hybrid v1.11 and newer it is usuallyENV_GROUP_NAME.Upgrade the chart:
Note: For installations migrated fromhelm upgradeENV_GROUP_RELEASE_NAME apigee-virtualhost/ \ --install \ --namespaceapigee \ --set envgroup=ENV_GROUP_NAME \ -fOVERRIDES_FILE
apigeectlto Helm, use: Note:ENV_GROUP_RELEASE_NAME must be unique within thehelm upgradeENV_GROUP_RELEASE_NAME apigee-virtualhost/ \ --install \ --namespaceapigee \ --set envgroup=ENV_GROUP_NAME \ --force \ -fOVERRIDES_FILE
apigeenamespace.For example, if you have a anenvironment named
prodand anenvironment group namedprod, set the value ofENV_GROUP_RELEASE_NAME to something unique, likeprod-envgroup. - Check the state of the ApigeeRoute (AR).
Installing the
virtualhostscreates ApigeeRouteConfig (ARC) which internally creates ApigeeRoute (AR) once the Apigee watcher pulls env group related details from the control plane. Therefore, check that the corresponding AR's state is running:kubectl -n apigee get arc
NAME STATE AGEapigee-org1-dev-egroup 2d
kubectl -n apigee get ar
NAME STATE AGEapigee-org1-dev-egroup-xxxxxx running 2d
- You must upgrade one environment group (virtualhost) at a time. Specify the environment group with
Install 1.11.2-hotfix.3
- Make the following change to youroverrides.yaml file to enable
runtimeto use the correct tag,1.11.2-hotfix.3:runtime: image: url: "gcr.io/apigee-release/hybrid/apigee-runtime" tag: "1.11.2-hotfix.3"
- Update the
apigee-envchart with thehelm upgradecommand and your current overrides file.Repeat this for every environment.
helm upgradeENV_RELEASE_NAME apigee-env/ \ --namespaceAPIGEE_NAMESPACE \ --set envENV_NAME \ --atomic \ -fOVERRIDES_FILE
apigeectl
- Store the latest version number in a variable using the following command:
Linux
export VERSION=$(curl -s \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt?ignoreCache=1)
Mac OS
export VERSION=$(curl -s \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt)
Windows
for /f "tokens=*" %a in ('curl -s ^ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt') ^do set VERSION=%a - Check that the variable was populated with a version number using the following command. If you want to use a different version, you can save that in an environment variable instead.
echo $VERSION
You should see the latest Apigee hybrid version:
1.11.2
- Be sure you are in the hybrid base directory (the parent of the directory where the
apigeectlexecutable file is located):cd$APIGEECTL_HOME/..
Download the release package for your operating system using the following command. Be sure to select your platform in the following table:
Linux
Linux 64 bit:
curl -LO \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/$VERSION/apigeectl_linux_64.tar.gz
Mac OS
Mac 64 bit:
curl -LO \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/$VERSION/apigeectl_mac_64.tar.gz
Windows
Windows 64 bit:
curl -LO ^ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/%VERSION%/apigeectl_windows_64.zip
- Rename your current
apigeectl/directory to a backup directory name. For example:Linux
mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.10/
Mac OS
mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.10/
Windows
rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.10
Extract the downloaded gzip file contents into your hybrid base directory. The hybrid base directory is the directory where the renamed
apigeectl-v1.10directory is located:Linux
tar xvzffilename.tar.gz -C ./
Mac OS
tar xvzffilename.tar.gz -C ./
Windows
tar xvzffilename.zip -C ./
The tar contents are, by default, expanded into a directory with the version and platform in its name. For example:
./apigeectl_1.11.2-xxxxxxx_linux_64. Rename that directory toapigeectlusing the following command:Linux
mvapigeectl_1.11.2-xxxxxxx_linux_64 apigeectl
Mac OS
mvapigeectl_1.11.2-xxxxxxx_mac_64 apigeectl
Windows
renameapigeectl_1.11.2-xxxxxxx_windows_64 apigeectl
- Change to the
apigeectldirectory:cd ./apigeectl
This directory is the
apigeectlhome directory. It is where theapigeectlexecutable command is located. - These instructions use the environment variable
$APIGEECTL_HOMEfor the directory in your file system where theapigeectlutility is installed. If needed, change directory into yourapigeectldirectory and define the variable with the following command:Linux
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Mac OS
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Windows
set APIGEECTL_HOME=%CD%
echo %APIGEECTL_HOME%
- Verify the version of
apigeectlwith theversioncommand:./apigeectl version
Version:1.11.2
- Create a
hybrid-base-directory/hybrid-filesdirectory, and then move into it. Thehybrid-filesdirectory is where configuration files such as the overrides file, certs, and service accounts are located. For example:Linux
mkdir$APIGEECTL_HOME/../hybrid-files
cd$APIGEECTL_HOME/../hybrid-files
Mac OS
mkdir$APIGEECTL_HOME/../hybrid-files
cd$APIGEECTL_HOME/../hybrid-files
Windows
mkdir%APIGEECTL_HOME%/../hybrid-files
cd%APIGEECTL_HOME%/../hybrid-files
- Verify that
kubectlis set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid.kubectl config get-contexts | grep \*
- In the
hybrid-filesdirectory:- Update the following symbolic links to
$APIGEECTL_HOME. These links allow you to run the newly installedapigeectlcommand from inside thehybrid-filesdirectory:ln -nfs
$APIGEECTL_HOME/tools toolsln -nfs$APIGEECTL_HOME/config configln -nfs$APIGEECTL_HOME/templates templatesln -nfs$APIGEECTL_HOME/plugins plugins - To check that the symlinks were created correctly, execute the following command and make sure the link paths point to the correct locations:
ls -l | grep ^l
- Update the following symbolic links to
- Do a dry run initialization to check for errors:
${APIGEECTL_HOME}/apigeectl init -fOVERRIDES_FILE --dry-run=clientWhereOVERRIDES_FILE is the name of your overrides file, for example
Tip: You can replace./overrides/overrides.yaml.OVERRIDES_FILEin the code sample above with the name and path to your overrides file, and every instance on this page will be replaced. - If there are no errors, initialize hybrid 1.11.2:
Note:$APIGEECTL_HOME/apigeectl init -fOVERRIDES_FILE
apigeectlinstalls and configures Apigee ingress gateway when you runapigeectl init. - Check the initialization status:
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
On success, the output says:
All containers ready.kubectl describe apigeeds -n apigee
In the output, look for
State: running. - Check for errors with a dry run of the
applycommand using the--dry-runflag:$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --dry-run=client
- If there are no errors, apply your overrides. Select and follow the instructions for production environments or non-prod environments, depending on your installation.
Production
For production environments, upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.
- Be sure you are in the
hybrid-filesdirectory. - Apply your overrides to upgrade Cassandra:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --datastore
- Check completion:
Tip: If$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
check-readyfails, you can get more information about your pods with:kubectl -nNAMESPACE get pods
WhereNAMESPACE is your Apigee hybrid namespace.
Proceed to the next step only when the pods are ready.
- Apply your overrides to upgrade Telemetry components and check completion:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --telemetry
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
- Bring up Redis components:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --redis
- Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --org
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
- Apply your overrides to upgrade your environments. You have two choices:
- Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --envENV_NAME
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
WhereENV_NAME is the name of the environment you are upgrading.
- All environments at one time: Apply your overrides to all environments at once and check completion:
$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --all-envs
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
- Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
- Apply your overrides to upgrade the
virtualhostscomponents and check completion:$APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --settings virtualhosts
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
Non-prod
In most non-production, demo, or experimental environments, you can apply the overrides to all components at once. If your non-production environment is large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments.
- Be sure you are in the
hybrid-filesdirectory. $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE
- Check the status:
$APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
- Be sure you are in the
Install 1.11.2-hotfix.3
- Make the following change to youroverrides.yaml file to enable
runtimeto use the correct tag,1.11.2-hotfix.3:runtime: image: url: "gcr.io/apigee-release/hybrid/apigee-runtime" tag: "1.11.2-hotfix.3"
- Install the hotfix with
apigeectl initusing your updated overrides file.Dry run:
${APIGEECTL_HOME}/apigeectl init -fOVERRIDES_FILE --dry-run=clientInitialize:
${APIGEECTL_HOME}/apigeectl init -fOVERRIDES_FILE - Apply the hotfix with
apigeectl apply.Dry run:
${APIGEECTL_HOME}/apigeectl apply -fOVERRIDES_FILE --all-envs --dry-run=clientApply:
${APIGEECTL_HOME}/apigeectl apply -fOVERRIDES_FILE --all-envs
Validate policies after upgrade to 1.11.2-hotfix.3
Use this procedure to validate the behavior of theJavaCallout policy after upgrading to Hybrid 1.11.2-hotfix.3.
- Check whether the Java JAR files request unnecessary permissions.
After the policy is deployed, check the runtime logs to see if the following log message is present:
"Failed to load and initialize class ...". If you observe this message, it suggests that the deployed JAR requested unnecessary permissions. To resolve this issue, investigate the Java code and update the JAR file. - Investigate and update the Java code.
Review any Java code (including dependencies) to identify the cause of potentially unallowed operations. When found, modify the source code as required.
- Test policies with the security check enabled.
In anon-production environment, enable the security check flag and redeploy your policies with an updated JAR. To set the flag:
- In the
apigee-env/values.yamlfile, setconf_security-secure.constructor.onlytotrueunderruntime:cwcAppend:. For example:# Apigee Runtimeruntime:cwcAppend:conf_security-secure.constructor.only:true
- Update the
apigee-envchart for the environment to apply the change. For example:helmupgradeENV_RELEASE_NAMEapigee-env/\--install\--namespaceAPIGEE_NAMESPACE\--setenv=ENV_NAME\-fOVERRIDES_FILE
ENV_RELEASE_NAME is a name used to keep track of installation and upgrades of the
apigee-envchart. This name must be unique from the other Helm release names in your installation. Usually this is the same asENV_NAME. However, if your environment has the same name as your environment group, you must use different release names for the environment and environment group, for exampledev-env-releaseanddev-envgroup-release. For more information on releases in Helm, seeThree big concepts in the Helm documentation.
If the log message
"Failed to load and initialize class ..."is still present, continue modifying and testing the JAR until the log message no longer appears. - In the
- Enable the security check in the production environment.
After you have thoroughly tested and verified the JAR file in the non-production environment, enable the security check in your production environment by setting the flag
conf_security-secure.constructor.onlytotrueand updating theapigee-envchart for the production environment to apply the change.
Rolling back an upgrade
Follow these steps to roll back a previous upgrade:
Helm
To rollback to the previous version, use the charts charts and the overrides file from the previous installation.
- Create the following environment variable:
- PREVIOUS_HELM_CHARTS_HOME: The directory where the previous Apigee hybrid Helm charts are installed. This is the version you are rolling back to.
- Roll back the virtualhosts. Repeat the following command for each environment group mentioned in the overrides file.
helm upgradeENV_GROUP_RELEASE_NAME $PREVIOUS_HELM_CHARTS_HOME/apigee-virtualhost/ \ --namespaceapigee \ --atomic \ --set envgroup=ENV_GROUP_NAME \ -fPREVIOUS_OVERRIDES_FILE
ENV_GROUP_RELEASE_NAME is the name with which you previously installed the
apigee-virtualhostchart. In hybrid v1.10, it is usuallyapigee-virtualhost-ENV_GROUP_NAME. In Hybrid v1.11 and newer it is usuallyENV_GROUP_NAME. - Roll back Envs. Repeat the following command for each environment mentioned in the overrides file.
helm upgrade apigee-env-ENV_NAME $PREVIOUS_HELM_CHARTS_HOME/apigee-env/ \ --install \ --namespaceapigee \ --atomic \ --set env=ENV_NAME \ -fPREVIOUS_OVERRIDES_FILE
ENV_RELEASE_NAME is the name with which you previously installed the
apigee-envchart. In hybrid v1.10, it is usuallyapigee-env-ENV_NAME. In Hybrid v1.11 and newer it is usuallyENV_NAME. - Roll back Org:
helm upgradeORG_NAME $PREVIOUS_HELM_CHARTS_HOME/apigee-org/ \ --install \ --namespaceapigee \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back the Ingress Manager:
helm upgrade ingress-manager $PREVIOUS_HELM_CHARTS_HOME/apigee-ingress-manager/ \ --install \ --namespaceapigee \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back Redis:
helm upgrade redis $PREVIOUS_HELM_CHARTS_HOME/apigee-redis/ \ --install \ --namespaceapigee \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back Apigee Telemetry:
helm upgrade telemetry $PREVIOUS_HELM_CHARTS_HOME/apigee-telemetry/ \ --install \ --namespaceapigee \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back Apigee Datastore (the Cassandra database component):
helm upgrade datastore $PREVIOUS_HELM_CHARTS_HOME/apigee-datastore/ \ --install \ --namespaceapigee \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back the Apigee Controller:
helm upgrade operator $PREVIOUS_HELM_CHARTS_HOME/apigee-operator/ \ --install \ --namespace apigee-system \ --atomic \ -fPREVIOUS_OVERRIDES_FILE
- Roll back the Apigee hybrid CRDs:
kubectl apply -k $PREVIOUS_HELM_CHARTS_HOME/apigee-operator/etc/crds/default/ --server-side --force-conflicts --validate=false
apigeectl
- Clean up completed jobs for the hybrid runtime namespace, whereNAMESPACE is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace is
apigee:kubectl delete job -nNAMESPACE \ $(kubectl get job -nNAMESPACE \ -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}') - Clean up completed jobs for the
apigee-systemnamespace:kubectl delete job -n apigee-system \ $(kubectl get job -n apigee-system \ -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}') - Change the
APIGEECTL_HOMEvariable to point to the directory that contains the previous version ofapigeectl. For example:export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
- Run the following commands in the root directory of the installation you want to roll back to. Be sure to use the original overrides file for the version you wish to roll back to:
- In the hybrid-files directory, run
apigeectl apply:$APIGEECTL_HOME/apigeectl apply -fORIGINAL_OVERRIDES_FILEWhereORIGINAL_OVERRIDES_FILE is the relative path and filename of the overrides file for your previous version hybrid installation, for example,
./overrides/overrides1.10.yaml. - Check the status of your pods:
kubectl -nNAMESPACE get pods
WhereNAMESPACE is your Apigee hybrid namespace.
- Check the status of
apigeeds:kubectl describe apigeeds -n apigee
Your output should look something like:
Status:CassandraDataReplication:CassandraPodIps:10.8.2.204CassandraReadyReplicas:1Components:Cassandra:LastSuccessfullyReleasedVersion:Revision:v1-f8aa9a82b9f69613Version:v1Replicas:Available:1Ready:1Total:1Updated:1State:runningScaling:InProgress:falseOperation:RequestedReplicas:0State:running
Proceed to the next step only when the
apigeedspod is running. - Run the following command to make note of what your new replica count values will be for the message processor after the upgrade. If these values do not match what you have set previously, change the values in your overrides file to match your previous configuration.
apigeectl apply -fORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2
Your output should look something like:
autoScaler: minReplicas: 2 maxReplicas: 10
- Run
apigeectl init:$APIGEECTL_HOME/apigeectl init -fORIGINAL_OVERRIDES_FILE
- In the hybrid-files directory, run
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-02-19 UTC.