This procedure covers upgrading from Apigee hybrid version 1.15.x to Apigee hybrid version 1.16.0.

Changes from Apigee hybrid v1.15

Please note the following changes:

• Seccomp profiles: Starting inversion 1.16, Apigee hybrid now offers the capability to apply Seccomp Profiles to your runtime components, significantly enhancing the security posture of your deployment. This feature allows Apigee administrators and security teams to restrict the system calls a containerized process can make to the host's kernel. By limiting a container to only the necessary syscalls, you can:
  • Enhance Security: Mitigate the risk of container breakouts and privilege escalation.
  • Enforce Least Privilege: Ensure components only have access to the exact system calls required for their operation.
  • Meet Compliance: Provide a critical control for meeting stringent security compliance requirements.
  For more information, seeConfigure Seccomp profiles for pod security.
• UDCA in Apigee hybrid removal: In Apigee hybridversion 1.16, the Unified Data Collection Agent (UDCA) component has been removed. The responsibilities of sending analytics, trace, and deployment status data to the Apigee control plane are now handled using aGoogle Cloud Pub/Sub based data pipeline. Using the Pub/Sub based data pipeline has been the default data collection mechanism since Apigee hybridversion 1.14.0.
• apigee-guardrails service account: In v1.16.0, Apigee Hybrid introduces anapigee-guardrails Google IAM service account. This is used by theapigee-operator chart during installation to check that all needed APIs are enabled in your project.

  See:

  • Diagnosing issues with guardrails
  • About service accounts:apigee-guardrails
  • create-service-account
  • Installation Part 2: Step 4: Create service accounts
  • Installation Part 2: Step 5: Set up service account authentication
• Support for cert-manager release 1.18 and 1.19: Apigee hybrid v1.16 supports cert-managerrelease 1.18 andrelease 1.19. In cert-manager release 1.18, there is a change to the default value ofCertificate.Spec.PrivateKey.rotationPolicy that can impact traffic. If you are upgrading from a previous version of Apigee hybrid, and you are uprading to cert-manager release 1.18 or above, follow theUpgrade cert-manager prodedure in this guide.
  Note: This does not affect fresh installations of Apigee hybrid v1.6.

For additional information about features in Hybrid version 1.16, see theApigee hybrid v1.16.0 release notes.

Prerequisites

Before upgrading to hybrid version 1.16, make sure your installation meets the following requirements:

• If your hybrid installation is running a version older than v1.15, you must upgrade to version 1.15 before upgrading to v1.16. SeeUpgrading Apigee hybrid to version 1.15.
• Helm version v3.14.2+.
• kubectl: A supported version ofkubectl appropriate for your Kubernetes platform version. SeeSupported platforms and versions:kubectl.
• cert-manager: A supported version of cert-manager. SeeSupported platforms and versions: cert-manager. If needed, you will upgrade cert-manager in thePrepare to upgrade to version 1.16 section below.

Before you upgrade to 1.16.0 - limitations and important notes

• Apigee hybrid 1.16.0 introduces a new enhanced per-environment proxy limit that lets you deploy more proxies and shared flows in a single environment. SeeLimits: API Proxies to understand the limits on the number of proxies and shared flows you may deploy per environment. This feature is available only on newly created hybrid organizations, and cannot be applied to upgraded orgs. To use this feature, perform a fresh installation of hybrid 1.16.0, and create a new organization.

  This feature is available exclusively as part of the2024 subscription plan, and is subject to the entitlements granted under that subscription. SeeEnhanced per-environment proxy limits to learn more about this feature.

• Upgrading to Apigee hybrid version 1.16 may require downtime.

  When upgrading the Apigee controller to version 1.16.0, all Apigee deployments undergo a rolling restart. To minimize downtime in production hybrid environments during a rolling restart, make sure you are running at least two clusters (in the same or different region/data center). Divert all production traffic to a single cluster and take the cluster you are about to upgrade offline, and then proceed with the upgrade process. Repeat the process for each cluster.

  Apigee recommends that once you begin the upgrade, you should upgrade all clusters as soon as possible to reduce the chances of production impact. There is no time limit on when all remaining clusters must be upgraded after the first one is upgraded. However, until all remaining clusters are upgraded Cassandra backup and restore cannot work with mixed versions. For example, a backup from Hybrid 1.15 cannot be used to restore a Hybrid 1.16 instance.

• Management plane changes do not need to be fully suspended during an upgrade. Any required temporary suspensions to management plane changes are noted in the upgrade instructions below.

Upgrading to version 1.16.0 overview

The procedures for upgrading Apigee hybrid are organized in the following sections:

1. Prepare to upgrade.
2. Install hybrid runtime version 1.16.0.

Prepare to upgrade to version 1.16

Back up your hybrid installation

1. These instructions use the environment variableAPIGEE_HELM_CHARTS_HOME for the directory in your file system where you have installed the Helm charts. If needed, change directory into this directory and define the variable with the following command:

   Linux

   export APIGEE_HELM_CHARTS_HOME=$PWDecho$APIGEE_HELM_CHARTS_HOME

   Mac OS

   export APIGEE_HELM_CHARTS_HOME=$PWDecho$APIGEE_HELM_CHARTS_HOME

   Windows

   set APIGEE_HELM_CHARTS_HOME=%CD%echo %APIGEE_HELM_CHARTS_HOME%

2. Make a backup copy of your version 1.15$APIGEE_HELM_CHARTS_HOME/ directory. You can use any backup process. For example, you can create atar file of your entire directory with:

   tar -czvf$APIGEE_HELM_CHARTS_HOME/../apigee-helm-charts-v1.15-backup.tar.gz$APIGEE_HELM_CHARTS_HOME

3. Back up your Cassandra database following the instructions inCassandra backup and recovery.
4. Make sure that your TLS certificate and key files (.crt,.key, and/or.pem) reside in the$APIGEE_HELM_CHARTS_HOME/apigee-virtualhost/ directory.

Upgrade your Kubernetes version

Check your Kubernetes platform version and if needed, upgrade your Kubernetes platform to a version that is supported by both hybrid 1.15 and hybrid 1.16. Follow your platform's documentation if you need help.

Click to expand a list of supported platforms

Supported versions

	1.14	1.15	1.16
GKE on Google Cloud	1.28.x 1.29.x 1.30.x 1.31.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
GKE on AWS	1.28.x 1.29.x(≥ 1.12.1)(8) 1.30.x 1.31.x	1.29.x(≥ 1.12.1)(8) 1.30.x 1.31.x 1.32.x	1.31.x 1.32.x 1.33.x
GKE on Azure	1.28.x 1.29.x(≥ 1.12.1)(8) 1.30.x 1.31.x	1.29.x(≥ 1.12.1)(8) 1.30.x 1.31.x 1.32.x	1.31.x 1.32.x 1.33.x
Google Distributed Cloud (software only) on VMware(3)	1.28.x 1.29.x 1.30.x 1.31.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
Google Distributed Cloud (software only) on bare metal	1.28.x 1.29.x 1.30.x 1.31.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
EKS	1.28.x 1.29.x 1.30.x 1.31.x 1.32.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
AKS	1.28.x 1.29.x 1.30.x 1.31.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
OpenShift(7)	4.13 4.14 4.15 4.16 4.17	4.16 4.17 4.18	4.18 4.19 4.20
Rancher Kubernetes Engine (RKE)	v1.27.x 1.28.x 1.29.x 1.30.x 1.31.x	v1.28.x 1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.31.x 1.32.x 1.33.x
VMware Tanzu	v1.26.x	v1.26.x	v1.26.x
Components	1.14	1.15	1.16
Cloud Service Mesh	1.22.x(2)	1.22.x(2)	1.22.x(2)
JDK	JDK 11	JDK 11	JDK 11
cert-manager	1.15.x(8) 1.16.x(8) 1.17.x(8)	1.16.x(8) 1.17.x(8)	1.16.x(8) 1.17.x(8) 1.18.x(9) 1.19.x(9)
Cassandra	4.0	4.0	4.0
Kubernetes	1.28.x 1.29.x 1.30.x 1.31.x 1.32.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.30.x 1.31.x 1.32.x 1.33.x
kubectl	1.28.x 1.29.x 1.30.x 1.31.x 1.32.x	1.29.x 1.30.x 1.31.x 1.32.x 1.33.x	1.30.x 1.31.x 1.32.x 1.33.x
Helm	3.14.2 through 3.x.x	3.14.2 through 3.x.x	3.17.0 through 3.x.x
Secret Store CSI driver	1.4.6+	1.4.6+	1.4.6+
Vault	1.17.2	1.17.2	1.17.2
(1) The official EOL dates for Apigee hybrid versions 1.13 and older have been reached. Regular monthly patches are no longer available. These releases are no longer officially supported except for customers with explicit and official exceptions for continued support. Other customers must upgrade. (2) Cloud Service Mesh is automatically installed with Apigee hybrid 1.9 and newer. (3) Vault is not certified on Google Distributed Cloud for VMware. (4) Support available with Apigee hybrid version 1.10.5 and newer. (5) Support available with Apigee hybrid version 1.11.2 and newer. (6) Support available with Apigee hybrid version 1.12.1 and newer. (7) Apigee hybrid is tested and certified on OpenShift using the Kubernetes version bundled with each specific OCP version. (8) Some versions of cert-manager have an issue where the webhook TLS server may fail to automatically renew its CA certificate. To avoid this, Apigee recommends using: Hybrid v1.13: use cert-manager version1.15.3+. Hybrid v1.14: use cert-manager versions1.15.3+ or1.16.0+. Hybrid v1.15: use cert-manager versions1.16.0+ or1.17.2+. Hybrid v1.16: use cert-manager versions1.16.0+,1.17.2+,1.18.0+ or1.19.0+. SeeKnown Issue 465834046 for cert-manager releases 1.18 and 1.19. (9) cert-manager release 1.18 introduces a change to the default private key rotation policy that may cause traffic interruptions. SeeKnown Issue 465834046 for a workaround.

Note: Hybrid installations are not currently supported on Autopilot clusters.

Unsupported versions

	1.6(1)	1.7(1)	1.8(1)	1.9(1)	1.10(1)	1.11(1)	1.12(1)	1.13(1)
GKE on Google Cloud	1.19.x 1.20.x 1.21.x	1.20.x 1.21.x 1.22.x(≥ 1.7.2) 1.23.x(≥ 1.7.2)	1.21.x(≤ 1.8.3) 1.22.x(≤ 1.8.3) 1.23.x(≤ 1.8.4) 1.24.x(≥ 1.8.4) 1.25.x(≥ 1.8.4)	1.23.x 1.24.x 1.25.x 1.26.x(≥ 1.9.2)	1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	1.27.x 1.28.x 1.29.x 1.30.x
GKE on AWS	1.7.x 1.8.x 1.9.3+ 1.10.x	1.9.x 1.10.x 1.12.x(≥ 1.7.2)	1.10.x 1.11.x 1.12.x (K8s v1.23.x) 1.13.x (K8s v1.24.x) 1.14.x (K8s v1.25.x)	1.12.x 1.13.x (K8s v1.24.x) 1.14.x (K8s v1.25.x)	1.13.x (K8s v1.24.x) 1.14.x (K8s v1.25.x) 1.26.x(3) 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.14.x (K8s v1.25.x) 1.26.x(3) 1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.26.x(3) 1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8) 1.30.x
GKE on Azure	1.8.x	1.9.x 1.10.x 1.12.x(≥ 1.7.2)	1.10.x 1.11.x 1.12.x 1.13.x 1.14.x	1.12.x 1.13.x 1.14.x	1.13.x 1.14.x 1.26.x(3) 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.14.x 1.26.x(3) 1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.26.x(3) 1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8) 1.30.x
Google Distributed Cloud (software only) on VMware(3)	1.7.x 1.8.x 1.9.3+ 1.10.x	1.9.x 1.10.x 1.11.x(3)(≥ 1.7.2) 1.12.x(≥ 1.7.2)	1.10.x 1.11.x 1.12.x 1.13.x 1.14.x 1.15.x (K8s v1.26.x)	1.12.x 1.13.x 1.14.x 1.15.x (K8s v1.26.x)	1.13.x 1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x 1.29.x(≥ 1.12.1)(8)	1.16.x (K8s v1.27.x) 1.28.x 1.29.x 1.30.x
Google Distributed Cloud (software only) on bare metal	1.7.x 1.8.2+ 1.9.3+ 1.10.x	1.9.x 1.10.x 1.11.x(≥ 1.7.2) 1.12.x(≥ 1.7.2)	1.10.x 1.11.x 1.12.x 1.13.x 1.14.x 1.15.x	1.12.x 1.13.x 1.14.x 1.15.x	1.13.x 1.14.x (K8s v1.25.x) 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.14.x 1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.15.x (K8s v1.26.x) 1.16.x (K8s v1.27.x) 1.28.x 1.29.x(≥ 1.12.1)(8)	1.16.x (K8s v1.27.x) 1.28.x 1.29.x 1.30.x
EKS	1.19.x 1.20.x 1.21.x	1.21.x 1.22.x(≥ 1.7.2) 1.23.x(≥ 1.7.2)	1.22.x(≤ 1.8.3) 1.23.x(≤ 1.8.4) 1.24.x(≥ 1.8.4) 1.25.x(≥ 1.8.4)	1.23.x 1.24.x 1.25.x 1.26.x	1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	1.27.x 1.28.x 1.29.x 1.30.x
AKS	1.19.x 1.20.x 1.21.x	1.21.x 1.22.x(≥ 1.7.2) 1.23.x(≥ 1.7.2)	1.22.x(≤ 1.8.3) 1.23.x(≤ 1.8.4) 1.24.x(≥ 1.8.4) 1.25.x(≥ 1.8.4)	1.23.x 1.24.x 1.25.x 1.26.x(≥ 1.9.2)	1.24.x 1.25.x 1.26.x 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	1.25.x 1.26.x 1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	1.26.x 1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	1.27.x 1.28.x 1.29.x 1.30.x
OpenShift(7)	4.6 4.7 4.8	4.7 4.8	4.8 4.9 4.10	4.10 4.11	4.11 4.12 4.14(≥ 1.10.5)(6) 4.15(≥ 1.10.5)(6)	4.12 4.13 4.14 4.15(≥ 1.11.2)(7) 4.16(≥ 1.11.2)(7)	4.12 4.13 4.14 4.15 4.16(≥ 1.12.1)(8)	4.12 4.13 4.14 4.15 4.16
Rancher Kubernetes Engine (RKE)	N/A	N/A	N/A	1.3.8	v1.26.2+rke2r1 1.27.x(≥ 1.10.5)(6) 1.28.x(≥ 1.10.5)(6)	v1.26.2+rke2r1 v1.27.x 1.28.x(≥ 1.11.2)(7) 1.29.x(≥ 1.11.2)(7)	v1.26.2+rke2r1 v1.27.x 1.28.x 1.29.x(≥ 1.12.1)(8)	v1.26.2+rke2r1 v1.27.x 1.28.x 1.29.x 1.30.x
VMware Tanzu	N/A	N/A	N/A	N/A	N/A	N/A	v1.26.x	v1.26.x
Components	1.6(1)	1.7(1)	1.8(1)	1.9(1)	1.10(1)	1.11(1)	1.12(1)	1.13(1)
Cloud Service Mesh	1.9.x 1.10.x 1.12.x 1.13.x	1.10.x 1.11.x 1.12.x 1.13.x(≥ 1.7.2) 1.14.x 1.15.x(≥ 1.7.6)	1.11.x 1.12.x 1.13.x 1.14.x 1.15.x	1.17.x(2)	1.17.x(2)	1.17.x(v1.11.0 - v1.11.1)(2) 1.18.x(≥ 1.11.2)(7)(2)	1.18.x(2)	1.19.x(2)
JDK	JDK 11	JDK 11	JDK 11	JDK 11	JDK 11	JDK 11	JDK 11	JDK 11
cert-manager	1.5.4	1.7.x	1.7.x 1.8.x 1.9.x 1.10.x 1.11.x	1.10.x 1.11.x	1.10.x 1.11.x 1.12.x	1.11.x 1.12.x 1.13.x	1.11.x 1.12.x 1.13.x	1.15.x(8) 1.16.x(8) 1.17.x(8)
Cassandra	3.11	3.11	3.11	3.11	3.11	3.11	4.0	4.0
Kubernetes				1.23.x 1.24.x 1.25.x	1.24.x 1.25.x 1.26.x	1.25.x 1.26.x 1.27.x	1.26.x 1.27.x 1.28.x 1.29.x	1.27.x 1.28.x 1.29.x 1.30.x
kubectl					1.24.x 1.25.x 1.26.x	1.25.x 1.26.x 1.27.x	1.26.x 1.27.x 1.28.x 1.29.x	1.27.x 1.28.x 1.29.x 1.30.x
Helm					3.10+	3.10+	3.14.2 through 3.x.x	3.14.2 through 3.x.x
Secret Store CSI driver						1.3.4+	1.4.1+	1.4.6+
Vault						1.13.x	1.15.2	1.15.2
(1) The official EOL dates for Apigee hybrid versions 1.13 and older have been reached. Regular monthly patches are no longer available. These releases are no longer officially supported except for customers with explicit and official exceptions for continued support. Other customers must upgrade. (2) Cloud Service Mesh is automatically installed with Apigee hybrid 1.9 and newer. (3) Vault is not certified on Google Distributed Cloud for VMware. (4) Support available with Apigee hybrid version 1.10.5 and newer. (5) Support available with Apigee hybrid version 1.11.2 and newer. (6) Support available with Apigee hybrid version 1.12.1 and newer. (7) Apigee hybrid is tested and certified on OpenShift using the Kubernetes version bundled with each specific OCP version. (8) Some versions of cert-manager have an issue where the webhook TLS server may fail to automatically renew its CA certificate. To avoid this, Apigee recommends using: Hybrid v1.13: use cert-manager version1.15.3+. Hybrid v1.14: use cert-manager versions1.15.3+ or1.16.0+. Hybrid v1.15: use cert-manager versions1.16.0+ or1.17.2+. Hybrid v1.16: use cert-manager versions1.16.0+,1.17.2+,1.18.0+ or1.19.0+. SeeKnown Issue 465834046 for cert-manager releases 1.18 and 1.19. (9) cert-manager release 1.18 introduces a change to the default private key rotation policy that may cause traffic interruptions. SeeKnown Issue 465834046 for a workaround.

Note: Hybrid installations are not currently supported on Autopilot clusters.

Pull the Apigee Helm charts.

Apigee hybrid charts are hosted inGoogle Artifact Registry:

oci://us-docker.pkg.dev/apigee-release/apigee-hybrid-helm-charts

Using thepull command, copy all of the Apigee hybrid Helm charts to your local storage with the following command:

exportCHART_REPO=oci://us-docker.pkg.dev/apigee-release/apigee-hybrid-helm-chartsexportCHART_VERSION=1.16.0-hotfix.1helm pull$CHART_REPO/apigee-operator --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-datastore --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-env --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-ingress-manager --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-org --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-redis --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-telemetry --version$CHART_VERSION --untarhelm pull$CHART_REPO/apigee-virtualhost --version$CHART_VERSION --untar

Editkustomization.yaml for a custom apigee namespace

If your Apigee namespace is notapigee, edit theapigee-operator/etc/crds/default/kustomization.yaml file and replace thenamespace value with your Apigee namespace.

apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomizationnamespace:APIGEE_NAMESPACE

If you are usingapigee as your namespace you do not need to edit the file.

Set up theapigee-guardrails service account

Starting with hybrid v1.16, theapigee-guardrails service account is required to upgrade theapigee-operator chart.

In the following procedure, select the type of service account authentication you are using.

1. Verify you can executecreate-service-account. If you have just downloaded the charts thecreate-service-account file might not be in an executable mode. In yourAPIGEE_HELM_CHARTS_HOME directory run the following command:

   ./apigee-operator/etc/tools/create-service-account --help

   If your output sayspermission denied you need to make the file executable, for example withchmod in Linux, MacOS, or UNIX or in the Windows Explorer or with theicacls command in Windows. For example:

   chmod +x ./apigee-operator/etc/tools/create-service-account

2. Create theapigee-guardrails service account:

   Kubernetes Secrets

   ./apigee-operator/etc/tools/create-service-account \  --env prod \  --profile apigee-guardrails \  --dir service-accounts

   This command creates theapigee-guardrails service account and downloads the key to theservice-accounts/ directory.

   JSON files

   ./apigee-operator/etc/tools/create-service-account \  --env prod \  --profile apigee-guardrails \  --dir ./apigee-operator/

   This command creates theapigee-guardrails service account and downloads the key to theapigee-operator/ chart directory.

   Vault

   ./apigee-operator/etc/tools/create-service-account \  --env prod \  --profile apigee-guardrails \  --dir service-accounts

   This command creates theapigee-guardrails service account and downloads the key to theservice-accounts/ directory.

   WIF for GKE

   ./apigee-operator/etc/tools/create-service-account \  --env prod \  --profile apigee-guardrails \  --dir service-accounts

   This command creates theapigee-guardrails service account and downloads the key to theapigee-operator/etc/tools/service-accounts/ directory. You do not need the downloaded key file and can delete it.

   WIF on other platforms

   ./apigee-operator/etc/tools/create-service-account \  --env prod \  --profile apigee-guardrails \  --dir service-accounts

   This command creates theapigee-guardrails service account and downloads the key to theservice-accounts/ directory.

3. Set up authentication for theapigee-guardrails service account:

   Kubernetes Secrets

   Create the Kubernetes secret using theapigee-guardrails service account key file in theservice-accounts/ directory:

   kubectl create secret generic apigee-guardrails-svc-account \    --from-file="client_secret.json=$APIGEE_HELM_CHARTS_HOME/service-accounts/$PROJECT_ID-apigee-guardrails.json" \    -n$APIGEE_NAMESPACE

   Add the following to youroverrides.yaml file:

   guardrails:serviceAccountRef:apigee-guardrails-svc-account

   JSON files

   Add the following to youroverrides.yaml file, using the path to theapigee-guardrails service account key file in theapigee-operator/ directory:

   guardrails:serviceAccountPath:$PROJECT_ID-apigee-guardrails.json

   Vault

   1. Update the Vault secretsecret/data/apigee/orgsakeys to add aguardrails entry with the contents of theapigee-guardrails service account key file.

      vault kv patch secret/apigee/orgsakeys guardrails="$(cat ./service-accounts/hybrid115-apigee-guardrails.json)"

   2. The Kubernetes service account (KSA) for guardrails is namedapigee-operator-guardrails-sa. Add the Guardrails KSA to the organization-specific service accounts bound to theapigee-orgsakeys role in Vault.
      1. Get the current list of KSAs bindings:

         vault read auth/kubernetes/role/apigee-orgsakeys

         The output should be in the following format:

         Key                                         Value---                                         -----alias_name_source                           serviceaccount_uidbound_service_account_namesBOUND_SERVICE_ACCOUNT_NAMESbound_service_account_namespace_selector    n/abound_service_account_namespacesAPIGEE_NAMESPACE

         In the output,BOUND_SERVICE_ACCOUNT_NAMES is a list of comma-separated service account names. Addapigee-operator-guardrails-sa to the list of names. for example (without the newlines added for readability):

         apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-myhybridorg-5b044c1,apigee-cassandra-schema-val-myhybridorg-5b044c1,apigee-cassandra-user-setup-myhybridorg-5b044c1,apigee-mart-myhybridorg-5b044c1,apigee-mint-task-scheduler-myhybridorg-5b044c1,apigee-connect-agent-myhybridorg-5b044c1,apigee-watcher-myhybridorg-5b044c1,apigee-metrics-apigee-telemetry,apigee-open-telemetry,apigee-synchronizer-myhybridorg-dev-ee52aca,apigee-runtime-telemetry-collector-apigee-telemetry,apigee-logger-apigee-e-myhybrridorg-dev-ee52aca,apigee-synchronizer-myhybridog-prod-2d0221c,apigee-runtime-myhybridorg-prod-2d0221c,apigee-operator-guardrails-sa

      2. Update the bindings to theapigee-orgsakeys role with the updated list of service account names:

         vault write auth/kubernetes/role/apigee-orgsakeys \  bound_service_account_names=UPDATED_BOUND_SERVICE_ACCOUNT_NAMES \  bound_service_account_namespaces=APIGEE_NAMESPACE \  policies=apigee-orgsakeys-auth \  ttl=1m

   3. Add "guardrails" to theSecretProviderClass
      1. Edit yourspc-org.yaml file.Tip: If you are need to recreate it, use the template inStoring service account keys in Hashicorp Vault: CreateSecretProviderClass objects.
      2. Underspec.parameters.objects, add a guardrails entry:

               - objectName: "guardrails"        secretPath: ""        secretKey: ""

         Tip: For scripted environments or to automate this change, you can use the command-line YAML processoryq.

         Command: To add the "guardrails" entry tospc-org.yaml in-place, use the following command:

         yq -i '.spec.parameters.objects |= (    from_yaml + [        {            "objectName": "guardrails",            "secretPath": "",            "secretKey": ""        }    ] | to_yaml)' spc-org.yaml

         This command parses the embedded YAML string in theobjects field, adds the new item to the list, and then writes the result back.

      3. Update yourSecretProviderClass:

         kubectl -nAPIGEE_NAMESPACE apply -f spc-org.yaml

   WIF for GKE

   The Kubernetes service account (KSA) for guardrails is namedapigee-operator-guardrails-sa. Create the binding for theapigee-guardrails Google service account (GSA) with the following command:

   gcloud iam service-accounts add-iam-policy-binding apigee-guardrails@$PROJECT_ID.iam.gserviceaccount.com \    --role roles/iam.workloadIdentityUser \    --member "serviceAccount:$PROJECT_ID.svc.id.goog[$APIGEE_NAMESPACE/apigee-operator-guardrails-sa]" \    --project$PROJECT_ID

   Add the following to youroverrides.yaml file:

   guardrails:gsa:apigee-guardrails@$PROJECT_ID.iam.gserviceaccount.com

   WIF on other platforms

   The Kubernetes service account (KSA) for guardrails is namedapigee-operator-guardrails-sa. You need to grant the guardrails KSA access to impersonate theapigee-guardrails Google service account (GSA), and configure your overrides to use a credential configuration file.

   1. Grant the KSA access to impersonate the GSA with the following command:

      Template

      gcloud iam service-accounts add-iam-policy-binding \  apigee-guardrails@$PROJECT_ID.iam.gserviceaccount.com \  --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/system:serviceaccount:APIGEE_NAMESPACE:apigee-operator-guardrails-sa" \  --role=roles/iam.workloadIdentityUser

      Example

      gcloud iam service-accounts add-iam-policy-binding \  apigee-guardrails@my-project.iam.gserviceaccount.com \  --member="principal://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/subject/system:serviceaccount:apigee:apigee-operator-guardrails-sa" \  --role=roles/iam.workloadIdentityUser

      Where:

      • PROJECT_ID: your Google Cloud project ID.
      • PROJECT_NUMBER: theproject number of the project where you created the workload identity pool.
      • POOL_ID: the workload identity pool ID.
      • APIGEE_NAMESPACE: The namespace where Apigee hybrid is installed.
   2. Create a credential configuration file for theapigee-guardrails service account:

      gcloud iam workload-identity-pools create-cred-config \  projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/WORKLOAD_PROVIDER_ID \  --service-account=apigee-guardrails@$PROJECT_ID.iam.gserviceaccount.com \  --credential-source-file=/var/run/service-account/token \  --credential-source-type=text \  --output-file=apigee-guardrails-credential-configuration.json

      WhereWORKLOAD_PROVIDER_ID is your workload identity pool provider ID.

   3. Configureapigee-guardrails to use Workload Identity Federation with one of the following methods:

      WIF: secrets

      1. Create a new Kubernetes secret using the credential source file for each credential configuration file.

         kubectl create secret -nAPIGEE_NAMESPACE generic guardrails-workload-identity-secret --from-file="client_secret.json=./apigee-guardrails-credential-configuration.json"

      2. Replace the value ofserviceAccountRef with the new secret:

         guardrails:serviceAccountRef:guardrails-workload-identity-secret

      WIF: files

      Move the generatedapigee-guardrails-credential-configuration.json file to yourapigee-operator/ chart directory.

      Add the following to youroverrides.yaml file:

      guardrails:serviceAccountPath:apigee-guardrails-credential-configuration.json

      WIF: Vault

      Update the service account key forguardrails in Vault with the corresponding credential source file:

      SAKEY=$(cat .apigee-guardrails-credential-configuration.json); kubectl -nAPIGEE_NAMESPACE exec vault-0 -- vault kv patch secret/apigee/orgsakeys guardrails="$SAKEY"

      SeeStoring service account keys in Hashicorp Vault for more information.

Upgrade cert-manager

1. Check the contents of yourapigee-ca certificate to see ifrotationPolicy is set:Tip: If you are using a custom cert-manager namesspace, replacecert-manager with the correct namespace in the following commands.

   kubectl get certificate apigee-ca -ncert-manager -o yaml

   Look for the values underspec.privateKey in the output:

   ...spec:  commonName: apigee-hybrid  duration: 87600h  isCA: true  issuerRef:    group: cert-manager.io    kind: ClusterIssuer    name: apigee-root-certificate-issuerprivateKey:    algorithm: ECDSA# Note: rotationPolicy would appear here if it is set.    size: 256  secretName: apigee-ca...

2. IfrotationPolicy is not set or if it is set toAlways, edit theapigee-ca certificate to set the value ofrotationPolicy toNever:
   1. Perform a dry run first:

      kubectl patch Certificate \  --dry-run=server \  -ncert-manager \  --type=json \  -p='[{"op": "replace", "path": "/spec/privateKey/rotationPolicy", "value": "Never"}]' \  -o=yaml \  apigee-ca

   2. Patch the certificate:

      kubectl patch Certificate \  -ncert-manager \  --type=json \  -p='[{"op": "replace", "path": "/spec/privateKey/rotationPolicy", "value": "Never"}]' \  -o=yaml \  apigee-ca

3. Verify that the value ofrotationPolicy is now set toNever:

   kubectl get certificate apigee-ca -ncert-manager -o yaml

   The output should look similar to the following:

   ...spec:  commonName: apigee-hybrid  duration: 87600h  isCA: true  issuerRef:    group: cert-manager.io    kind: ClusterIssuer    name: apigee-root-certificate-issuerprivateKey:    algorithm: ECDSArotationPolicy: Never    size: 256  secretName: apigee-ca...

4. Upgrade cert-manager. The following command will download and install cert-manager v1.19.2:

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml

   SeeSupported platforms and versions: cert-manager for a list of supported versions.

See:

• Known issue 465834046
• cert-manager Release 1.18: The default value ofCertificate.Spec.PrivateKey.rotationPolicy is nowAlways

Install the hybrid 1.16.0 runtime

1. If you have not, navigate into yourAPIGEE_HELM_CHARTS_HOME directory. Run the following commands from that directory.
2. Upgrade the Apigee Operator/Controller:Note: This step requires elevated cluster permissions. Runhelm -h orhelm upgrade -h for details.

   Dry run:

   helm upgrade operator apigee-operator/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgrade operator apigee-operator/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verify Apigee Operator installation:

   helm ls -nAPIGEE_NAMESPACE

   NAME       NAMESPACE       REVISION   UPDATED                                STATUS     CHART                   APP VERSIONoperator   apigee   3          2024-08-21 00:42:44.492009 -0800 PST   deployed   apigee-operator-1.16.0   1.16.0

   Verify it is up and running by checking its availability:

   kubectl -nAPIGEE_NAMESPACE get deploy apigee-controller-manager

   NAME                        READY   UP-TO-DATE   AVAILABLE   AGEapigee-controller-manager   1/1     1            1           7d20h

3. Upgrade the Apigee datastore:

   Dry run:

   helm upgrade datastore apigee-datastore/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgrade datastore apigee-datastore/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verifyapigeedatastore is up and running by checking its state:

   kubectl -nAPIGEE_NAMESPACE get apigeedatastore default

   NAME      STATE       AGEdefault   running    2d

4. Upgrade Apigee telemetry:

   Dry run:

   helm upgrade telemetry apigee-telemetry/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgrade telemetry apigee-telemetry/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verify it is up and running by checking its state:

   kubectl -nAPIGEE_NAMESPACE get apigeetelemetry apigee-telemetry

   NAME               STATE     AGEapigee-telemetry   running   2d

5. Upgrade Apigee Redis:

   Dry run:

   helm upgrade redis apigee-redis/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgrade redis apigee-redis/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verify it is up and running by checking its state:

   kubectl -nAPIGEE_NAMESPACE get apigeeredis default

   NAME      STATE     AGEdefault   running   2d

6. Upgrade Apigee ingress manager:

   Dry run:

   helm upgrade ingress-manager apigee-ingress-manager/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgrade ingress-manager apigee-ingress-manager/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verify it is up and running by checking its availability:

   kubectl -nAPIGEE_NAMESPACE get deployment apigee-ingressgateway-manager

   NAME                            READY   UP-TO-DATE   AVAILABLE   AGEapigee-ingressgateway-manager   2/2     2            2           2d

7. Upgrade the Apigee organization:

   Dry run:

   helm upgradeORG_NAME apigee-org/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE \  --dry-run=server

   Upgrade the chart:

   helm upgradeORG_NAME apigee-org/ \  --install \  --namespaceAPIGEE_NAMESPACE \  -fOVERRIDES_FILE

   Verify it is up and running by checking the state of the respective org:

   kubectl -nAPIGEE_NAMESPACE get apigeeorg

   NAME                      STATE     AGEapigee-my-org-my-env      running   2d

   Note: If the upgrade command fails with the errorForbidden: state: releasing, the existing components are still in releasing status. You can wait for the current update to complete and then retry. Check release status with the following command:

   kubectl get org -nAPIGEE_NAMESPACE

8. Upgrade the environment.

   You must install one environment at a time. Specify the environment with--set env=ENV_NAME.

   Dry run:

   helm upgradeENV_RELEASE_NAME apigee-env/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --set env=ENV_NAME \  -fOVERRIDES_FILE \  --dry-run=server

   • ENV_RELEASE_NAME is a name used to keep track of installation and upgrades of theapigee-env chart. This name must be unique from the other Helm release names in your installation. Usually this is the same asENV_NAME. However, if your environment has the same name as your environment group, you must use different release names for the environment and environment group, for exampledev-env-release anddev-envgroup-release. For more information on releases in Helm, seeThree big concepts in the Helm documentation.
   • ENV_NAME is the name of the environment you are upgrading.
   • OVERRIDES_FILE is your new overrides file for v.1.16.0

   Upgrade the chart:

   helm upgradeENV_RELEASE_NAME apigee-env/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --set env=ENV_NAME \  -fOVERRIDES_FILE

   Verify it is up and running by checking the state of the respective env:

   kubectl -nAPIGEE_NAMESPACE get apigeeenv

   NAME                          STATE       AGE   GATEWAYTYPEapigee-my-org-my-env          running     2d

Note: If the upgrade command fails with the errorForbidden: state: releasing, the existing components are still in releasing status. You can wait for the current update to complete and then retry. Check release status with the following command:

kubectl get env -nAPIGEE_NAMESPACE

9. Upgrade the environment groups (virtualhosts).
   1. You must upgrade one environment group (virtualhost) at a time. Specify the environment group with--set envgroup=ENV_GROUP_NAME. Repeat the following commands for each environment group mentioned in the overrides.yaml file:

      Dry run:

      helm upgradeENV_GROUP_RELEASE_NAME apigee-virtualhost/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --set envgroup=ENV_GROUP_NAME \  -fOVERRIDES_FILE \  --dry-run=server

      ENV_GROUP_RELEASE_NAME is the name with which you previously installed theapigee-virtualhost chart. It is usuallyENV_GROUP_NAME.

      Upgrade the chart:

      helm upgradeENV_GROUP_RELEASE_NAME apigee-virtualhost/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --set envgroup=ENV_GROUP_NAME \  -fOVERRIDES_FILE

      Note:ENV_GROUP_RELEASE_NAME must be unique within theapigee namespace.

      For example, if you have anenvironment namedprod and anenvironment group namedprod, set the value ofENV_GROUP_RELEASE_NAME to something unique, likeprod-envgroup.

   2. Check the state of the ApigeeRoute (AR).

      Installing thevirtualhosts creates ApigeeRouteConfig (ARC) which internally creates ApigeeRoute (AR) once the Apigee watcher pulls environment group-related details from the control plane. Therefore, check that the corresponding AR's state is running:

      kubectl -nAPIGEE_NAMESPACE get arc

      NAME                                STATE   AGEapigee-org1-dev-egroup                       2d

      kubectl -nAPIGEE_NAMESPACE get ar

      NAME                                        STATE     AGEapigee-org1-dev-egroup-123abc               running   2d

10. After you have verified all the installations are upgraded successfully, delete the olderapigee-operator release from theapigee-system namespace.
    1. Uninstall the oldoperator release:

       helm delete operator -n apigee-system

    2. Delete theapigee-system namespace:

       kubectl delete namespace apigee-system

11. Upgradeoperator again in your Apigee namespace to re-install the deleted cluster-scoped resources:

    helm upgrade operator apigee-operator/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --atomic \  -f overrides.yaml

Rolling back to a previous version

To roll back to the previous version, use the older chart version to roll back the upgrade process in the reverse order. Start withapigee-virtualhost and work your way back toapigee-operator, and then revert the CRDs.