Upgrading Apigee hybrid to version 1.8

You are currently viewing version 1.8 of the Apigee hybrid documentation.This version is end of life. You should upgrade to a newer version. For more information, seeSupported versions.
Note:This procedure covers upgrading from Apigee hybrid version 1.7.x or from previous releases of Apigee hybrid version 1.8.x to Apigee hybrid version 1.8.8.

Use the same procedures for minor version upgrades (for example version 1.7 to 1.8) and for patch release upgrades (for example 1.8.0 to 1.8.8).

If you are upgrading from Apigee hybrid version 1.6 or older, you must first upgrade to hybrid version 1.7 before upgrading to version 1.8.8. See the instructions forUpgrading Apigee hybrid to version 1.7.

If you are already on hybrid v1.8.0 and are looking to migrate to Apigee ingress gateway from Anthos Service Mesh, see Migrating to Apigee ingress gateway.

Introducing Apigee ingress gateway

Starting in version 1.8, Apigee hybrid offers a new feature to manage the ingress gateway for your hybrid installation,Apigee ingress gateway. Anthos Service Mesh is no longer a prerequisite for hybrid installation. With Apigee ingress gateway, Apigee will stop supplying routing configuration to Anthos Service Mesh. After the upgrade, you have to migrate the traffic to the new Apigee ingress gateway before you can start using the feature.

Apigee uses a small subset of Anthos Service Mesh features for the ingress gateway. Starting with hybrid version 1.8 Apigee hybrid includes an ingress gateway which is installed and upgraded as part of Apigee hybrid upgrades. Therefore you do not need to build expertise around Anthos Service Mesh to install, upgrade, and manage Apigee hybrid. Issues around ingress gateway versions and compatibility with Apigee hybrid releases are handled automatically.

Two scenarios for migrating are:

  • Multi-cluster or multi-region migration (recommended):

    Before switching to a new Ingress for Apigee, drain all the traffic to another cluster or region from the cluster you are migrating. This will give you time to test if the new Apigee ingress gateway is working as expected. Then shift the traffic back to the upgraded cluster.

  • In-place upgrade (not recommended in production environments):

    During the upgrade Apigee will bring up the new ingress gateway with an IP address you specify. You can then test if the new Apigee ingress gateway is working as expected, and then shift traffic to the new ingress. There might be downtime during this upgrade.

When upgrading Apigee hybrid to version 1.8, you must configure Apigee ingress gateway in your overrides file. After upgrading, you control which type of ingress gateway your clusters will use by directing the A or CNAME records at your registrar to the IP address for Apigee ingress gateway or for Anthos Service Mesh.

Caution: Because Apigee ingress gateway is a new feature which introduces significant change,Apigee recommends you upgrade and test a non-production installation first, before you upgrade aproduction installation.

Upgrading to version 1.8.8 overview

Upgrading to Apigee hybrid version 1.8 may require downtime.

When upgrading the Apigee controller to version 1.8.8, all Apigee deployments undergo a rolling restart. To minimize downtime in production hybrid environments during a rolling restart, make sure you are running at least two clusters (in the same or different region/data center). Divert all production traffic to a single cluster and take the cluster you are about to upgrade offline, and then proceed with the upgrade process. Repeat the process for each cluster.

Apigee recommends that you upgrade all clusters as soon as possible to reduce the chances of production impact. There is no time limit on when all remaining clusters must be upgraded after the first one is upgraded. However, until all remaining clusters are upgraded the following operations will be impacted:

  • Cassandra backup and restore cannot work with mixed versions. For example, a backup from Hybrid 1.7 cannot be used to restore a Hybrid 1.8 instance.
  • Cassandra data streaming will not work between mixed Hybrid versions. Therefore, your Cassandra clusters cannot scale horizontally.
  • Region expansion and decommissioning will be impacted, because these operations depend on Cassandra data streaming.

The procedures for upgrading Apigee hybrid are organized in the following sections:

  1. Prepare to upgrade.
  2. Install hybrid runtime version 1.8.8.
  3. For the ingress gateway, pick one of the following options:

Prerequisite

These upgrade instructions assume you have Apigee hybrid version 1.7.x or an earlier patch release of version 1.8.x installed and wish to upgrade it to version 1.8.8. If you are updating from an earlier version see the instructions forUpgrading Apigee hybrid to version 1.7.

If you prefer to continue using Anthos Service Mesh, you must ensure that Anthos Service Mesh is upgraded to a supported version. See theSupported platforms table for supported versions of Anthos Service Mesh.

Prepare to upgrade to version 1.8

Back up your hybrid installation (recommended)

  1. These instructions use the environment variableAPIGEECTL_HOME for the directory in your file system where you have installedapigeectl. If needed, change directory into yourapigeectl directory and define the variable with the following command:

    Linux

    export APIGEECTL_HOME=$PWDecho$APIGEECTL_HOME

    Mac OS

    export APIGEECTL_HOME=$PWDecho$APIGEECTL_HOME

    Windows

    set APIGEECTL_HOME=%CD%echo %APIGEECTL_HOME%
  2. Make a backup copy of your version 1.7$APIGEECTL_HOME/ directory. For example:
    tar -czvf$APIGEECTL_HOME/../apigeectl-v1.7-backup.tar.gz$APIGEECTL_HOME
  3. Back up your Cassandra database following the instructions inCassandra backup and recovery

Add theCloud Trace Agent role to the service account for the Apigee runtime. (Optional)

Optional: If you plan to useCloud trace and you have not already performed this step on your hybrid v1.7 installation, ensure your service account for your Apigee runtime services has theCloud Trace Agent Google role. (roles/cloudtrace.agent).

For production environments, this is usually theapigee-runtime service account. For non-production environments, this is usually theapigee-non-prod service account.

You can add the role in theCloud console > IAM & Admin > Service accounts UI or with the following commands:

  1. Get the email address for your service account with the following command:

    Production

    gcloud iam service-accounts list --filter "apigee-runtime"

    If it matches the patternapigee-runtime@$ORG_NAME.iam.gserviceaccount.com, you can use that pattern in the next step.

    Non-Prod

    gcloud iam service-accounts list --filter "apigee-non-prod"

    If it matches the patternapigee-non-prod@$ORG_NAME.iam.gserviceaccount.com, you can use that pattern in the next step.

  2. Assign theCloud Trace Agent role to the service account:

    Production

    gcloud projects add-iam-policy-binding$PROJECT_ID \    --member="serviceAccount:apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com" \    --role="roles/cloudtrace.agent"

    Non-Prod

    gcloud projects add-iam-policy-binding$PROJECT_ID \    --member="serviceAccount:apigee-non-prod@$PROJECT_ID.iam.gserviceaccount.com" \    --role="roles/cloudtrace.agent"

    Example

    gcloud projects add-iam-policy-binding hybrid-example-project \    --member="serviceAccount:apigee-runtime@hybrid-example-project.iam.gserviceaccount.com" \    --role="roles/cloudtrace.agent"

    Where:$PROJECT_ID is the name of the Google Cloud project where Apigee hybrid is installed.

    Note:If your installation uses a custom name for the service account, substitute that name forapigee-runtime orapigee-non-prod in the command.

Prepare to install Apigee ingress gateway

To install Apigee ingress gateway as part of the upgrade. You need to add the followingingressGateways property to your overrides file.

Syntax

ingressGateways:- name:INGRESS_NAME  replicaCountMin:REPLICAS_MIN  replicaCountMax:REPLICAS_MAX  resources:    requests:      cpu:CPU_COUNT_REQ      memory:MEMORY_REQ    limits:      cpu:CPU_COUNT_LIMIT      memory:MEMORY_LIMIT  svcAnnotations:  # optional. SeeKnown issue 243599452.SVC_ANNOTATIONS_KEY:SVC_ANNOTATIONS_VALUE  svcLoadBalancerIP:SVC_LOAD_BALANCER_IP # optional

Example

ingressGateways:- name: prod1  replicaCountMin: 2  replicaCountMax: 100  resources:    requests:      cpu: 1      memory: 1Gi    limits:      cpu: 2      memory: 2Gi

Make additional changes to your overrides file to enable or disable optional v1.8 features

Add the following properties to youroverrides.yaml file to enable new features in hybrid v1.8. These features are optional.

Install the hybrid 1.8.8 runtime

Caution:Do not create new environments during the upgrade process.
  1. Be sure you are in the hybrid base directory (the parent of the directory where theapigeectl executable file is located):
    cd$APIGEECTL_HOME/..

  2. Download the release package for your operating system using the following command. Be sure to select your platform in the following table:

    Linux

    Linux 64 bit:

    curl -LO \  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_linux_64.tar.gz

    Mac OS

    Mac 64 bit:

    curl -LO \  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_mac_64.tar.gz

    Windows

    Windows 64 bit:

    curl -LO ^  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_windows_64.zip
  3. Rename your currentapigeectl/ directory to a backup directory name. For example:

    Linux

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/

    Mac OS

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/

    Windows

    rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.7

  4. Extract the downloaded gzip file contents into your hybrid base directory. The hybrid base directory is the directory where the renamedapigeectl-v1.7 directory is located:

    Linux

    tar xvzffilename.tar.gz -C ./

    Mac OS

    tar xvzffilename.tar.gz -C ./

    Windows

    tar xvzffilename.zip -C ./

  5. The tar contents are, by default, expanded into a directory with the version and platform in its name. For example:./apigeectl_1.8.8-xxxxxxx_linux_64. Rename that directory toapigeectl using the following command:

    Linux

    mvapigeectl_1.8.8-xxxxxxx_linux_64 apigeectl

    Mac OS

    mvapigeectl_1.8.8-xxxxxxx_mac_64 apigeectl

    Windows

    renameapigeectl_1.8.8-xxxxxxx_windows_64 apigeectl
  6. Change to theapigeectl directory:
    cd ./apigeectl

    This directory is theapigeectl home directory. It is where theapigeectl executable command is located.

  7. These instructions use the environment variable$APIGEECTL_HOME for the directory in your file system where theapigeectl utility is installed. If needed, change directory into yourapigeectl directory and define the variable with the following command:

    Linux

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Mac OS

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Windows

    set APIGEECTL_HOME=%CD%
    echo %APIGEECTL_HOME%
  8. Verify the version ofapigeectl with theversion command:
    ./apigeectl version
    Version:1.8.8
  9. Move to thehybrid-base-directory/hybrid-files directory. Thehybrid-files directory is where configuration files such as the overrides file, certs, and service accounts are located. For example:
    cd$APIGEECTL_HOME/../hybrid-files
  10. Verify thatkubectl is set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid.
    kubectl config get-contexts | grep \*
  11. In thehybrid-files directory:
    1. Update the following symbolic links to$APIGEECTL_HOME. These links allow you to run the newly installedapigeectl command from inside thehybrid-files directory:
      ln -nfs$APIGEECTL_HOME/tools toolsln -nfs$APIGEECTL_HOME/config configln -nfs$APIGEECTL_HOME/templates templatesln -nfs$APIGEECTL_HOME/plugins plugins
    2. To check that the symlinks were created correctly, execute the following command and make sure the link paths point to the correct locations:
      ls -l | grep ^l
  12. Do a dry run initialization to check for errors:
    ${APIGEECTL_HOME}/apigeectl init -fOVERRIDES_FILE --dry-run=client

    WhereOVERRIDES_FILE is the name of your overrides file, for example./overrides/overrides.yaml.

    Tip: You can replaceOVERRIDES_FILE in the code sample above with the name and path to your overrides file, and every instance on this page will be replaced.
  13. If there are no errors, initialize hybrid 1.8.8. This command also installs and configures Apigee ingress gateway:
    $APIGEECTL_HOME/apigeectl init -fOVERRIDES_FILE
  14. Check the initialization status:
    $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE

    On success, the output says:All containers ready.

    Known issue: After you runinit and confirm thatcheck-ready succeeds, you may notice, if you view the pods, that the Cassandra schema validation is in an error state. This state is considered aknown issue (#245664917) and is harmless. You can safely ignore the error and go on to the next step:apigeectl apply.

    As a further check, you can also run this command to check ApigeeDataStore status:

    kubectl describe apigeeds -n apigee

    In the output, look forState: running.

  15. Check for errors with a dry run of theapply command using the--dry-run flag:
    $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --dry-run=client
  16. If there are no errors, apply your overrides. Select and follow the instructions for production environments or non-prod environments, depending on your installation.During the upgrade process each component will perform a rolling restart. Therefore, for production environments, it is best to apply the upgrade to one component at a time.

    Production

    For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.

    1. Be sure you are in thehybrid-files directory.
    2. Apply your overrides to upgrade Cassandra:
      $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --datastore
    3. Check completion:
      $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
      Tip: Ifcheck-ready fails, you can get more information about your pods with:
      kubectl -nNAMESPACE get pods

      WhereNAMESPACE is your Apigee hybrid namespace.

      Proceed to the next step only when the pods are ready.

    4. Apply your overrides to upgrade Telemetry components and check completion:
      $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --telemetry
      $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
    5. Bring up Redis components:
      $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --redis
    6. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
      $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --org
      $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
    7. Apply your overrides to upgrade your environments. You have two choices:
      • Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
        $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --envENV_NAME
        $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE

        WhereENV_NAME is the name of the environment you are upgrading.

      • All environments at one time: Apply your overrides to all environments at once and check completion:
        $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --all-envs
        $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE
    8. Apply your overrides to upgrade thevirtualhosts components and check completion:
      $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE --settings virtualhosts
      $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE

    Non-prod

    In most non-production, demo, or experimental environments, you can apply the overrides to all components at once. If your non-production environment is large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments.

    1. Be sure you are in thehybrid-files directory.
    2. $APIGEECTL_HOME/apigeectl apply -fOVERRIDES_FILE
    3. Check the status:
      $APIGEECTL_HOME/apigeectl check-ready -fOVERRIDES_FILE

Upgrade your Kubernetes version

Upgrade your Kubernetes platform to the versions supported by hybrid 1.8. Follow your platform's documentation if you need help.

Click to expand a list of supported platforms

Apigee hybrid versions

Platforms

 1.6not supported(3)1.7not supported(3)1.8not supported(3)1.91.10
Anthos (Google Cloud - GKE)1.19.x
1.20.x
1.21.x		1.20.x
1.21.x
1.22.x(≥ 1.7.2)
1.23.x(≥ 1.7.2)		1.21.x(≤ 1.8.3)
1.22.x(≤ 1.8.3)
1.23.x(≤ 1.8.4)
1.24.x(≥ 1.8.4)
1.25.x(≥ 1.8.4)		1.23.x
1.24.x
1.25.x
1.26.x(≥ 1.9.2)		1.24.x
1.25.x
1.26.x
Anthos (AWS)1.7.x
1.8.x
1.9.3+
1.10.x		1.9.x
1.10.x
1.12.x(≥ 1.7.2)		1.10.x
1.11.x
1.12.x
1.13.x
1.25(13)		1.12.x
1.13.x
1.25(13)		1.13.x
1.25(13)
1.26(13)
Anthos (Azure)1.8.x1.9.x
1.10.x
1.12.x(≥ 1.7.2)		1.10.x
1.11.x
1.12.x
1.13.x
1.14.x		1.12.x
1.13.x
1.14.x		1.13.x
1.14.x
1.15.x
Anthos (on-premises - VMware)(1)1.7.x
1.8.x
1.9.3+
1.10.x		1.9.xnot supported(4)
1.10.xnot supported(4)
1.11.xnot supported(4)(≥ 1.7.2)
1.12.xnot supported(4)(≥ 1.7.2)		1.10.xnot supported(4)
1.11.xnot supported(4)
1.12.xnot supported(4)(5)
1.13.x(5)
1.14.x(5)
1.15.x		1.12.xnot supported(4)
1.13.x
1.14.x
1.15.x		1.13.x
1.14.x
1.15.x
1.16.x
Anthos (Bare Metal)(1)1.7.x
1.8.2+
1.9.3+
1.10.x		1.9.xnot supported(4)
1.10.xnot supported(4)
1.11.xnot supported(4)(≥ 1.7.2)
1.12.xnot supported(4)(≥ 1.7.2)		1.10.xnot supported(4)
1.11.xnot supported(4)
1.12.xnot supported(4)(5)
1.13.x(5)
1.14.x(5)
1.15.x		1.12.xnot supported(4)
1.13.x
1.14.x
1.15.x		1.13.x
1.14.x
1.15.x
1.16.x
EKS(7)1.19.x
1.20.x
1.21.x		1.21.x
1.22.x(≥ 1.7.2)
1.23.x(≥ 1.7.2)		1.22.x(≤ 1.8.3)
1.23.x(≤ 1.8.4)
1.24.x(≥ 1.8.4)
1.25.x(≥ 1.8.4)		1.23.x
1.24.x
1.25.x
1.26.x(≥ 1.9.2)		1.24.x
1.25.x
1.26.x
AKS(7)1.19.x
1.20.x
1.21.x		1.21.x
1.22.x(≥ 1.7.2)
1.23.x(≥ 1.7.2)		1.22.x(≤ 1.8.3)
1.23.x(≤ 1.8.4)
1.24.x(≥ 1.8.4)
1.25.x(≥ 1.8.4)		1.23.x
1.24.x
1.25.x
1.26.x(≥ 1.9.2)		1.24.x
1.25.x
1.26.x
OpenShift(7)4.6
4.7
4.8		4.7
4.8		4.8
4.9
4.10		4.10
4.11		4.11
4.12
Rancher Kubernetes Engine (RKE)(7)N/AN/AN/A1.3.8v1.26.2+rke2r1

Components

1.6not supported(3)1.7not supported(3)1.81.91.10
Anthos Service Mesh (ASM)1.9.xnot supported
1.10.xnot supported
1.12.xnot supported
1.13.xnot supported		1.10.xnot supported
1.11.xnot supported
1.12.xnot supported
1.13.xnot supported(≥ 1.7.2)
1.14.xnot supported
1.15.x(≥ 1.7.6)		1.11.xnot supported
1.12.xnot supported
1.13.xnot supported
1.14.xnot supported
1.15.x		 1.17.x(11) 1.17.x(11)
JDKJDK 11JDK 11JDK 11JDK 11JDK 11
cert-manager1.5.41.7.x1.7.x
1.8.x
1.9.x
1.10.x
1.11.x		1.10.x
1.11.x		1.10.x
1.11.x
1.12.x
Cassandra3.11.103.11.103.11.103.11.103.11.10

(1) On Anthos versions 1.8, 1.9, and 1.10, follow the instructions in these documents to avoid conflict withcert-manager:

(2) Support available with Apigee hybrid version 1.7.2 and newer.

(3) The official EOL dates for Apigee hybrid versions 1.8 and older have been reached. Regular monthly patches are no longer available. These releases are no longer officially supported except for customers with explicit and official exceptions for continued support. Other customers must upgrade.

(4)Anthos on Bare Metal and VMWare versions 1.12 and earlier are out of support. See the Anthos on Bare Metal Version Support Policy and the Anthos clusters on VMware versions.

(5)Anthos on Bare Metal and VMWare requires ASM 1.14 or later. We recommend that you upgrade to hybrid v1.8 and switch to Apigee ingress gateway which no longer requires you to install ASM on your hybrid cluster.

(6) Support available with Apigee hybrid version 1.8.4 and newer.

(7) Attached clusters are not required for Apigee hybrid v1.8 and newer using Apigee ingressgateway..

(8) Not supported with Apigee hybrid version 1.8.4 and newer.

(9) Support available with Apigee hybrid version 1.7.6 and newer.

(10) Not supported with Apigee hybrid version 1.8.5 and newer.

(11) ASM is automatically installed with Apigee hybrid 1.9 and newer.

(12) Support available with Apigee hybrid version 1.9.2 and newer.

(13) Anthos clusters on AWS (Multi-Cloud) version numbers now reflect the Kubernetes versions. SeeAnthos version and upgrade support for version details and recommended patches.

About attached clusters

For Apigee hybrid versions 1.7.x and older, you must use Anthosattached clusters if you want to run Apigee hybrid in a multi-cloud context on Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or another supported third-party Kubernetes service provider. Cluster attachment allows Google to measure the usage of Anthos Service Mesh (ASM).Registering the third-party cluster is optional. Register only if you wish to view the attached cluster in the Google Cloud console. For more information, seeAttach third-party Kubernetes clusters to Google Cloud.

For Apigee hybrid version 1.8.x, Anthos attached clusters are required if you are using Anthos Service Mesh for your ingress gateway. If you are using Apigee ingress gateway, Anthos attached clusters are optional.

Note: Hybrid installations are not currently supported on GKE Autopilot.

Switch traffic from Anthos Service Mesh to Apigee ingress gateway

To switch your traffic to Apigee ingress gateway:

  1. Expose the Apigee ingress gateway. Follow the procedures inExpose Apigee ingress gateway.
  2. Test your new ingress gateway by calling a proxy. Ideally, test all crucial proxies you currently have deployed.
  3. To switch the traffic, update your DNS records to point to the IP address for your new Apigee ingress gateway. Depending on your DNS provider, you might be able to gradually shift traffic to the new endpoint.Tip:You can find the external IP address of Apigee ingress gateway with the following command:
    kubectl get svc -n apigee -l app=apigee-ingressgateway

    Your output should look something like:

    NAME                                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                      AGEapigee-ingressgateway-prod-hybrid-37a39bd   LoadBalancer   192.0.2.123233.252.0.123   15021:32049/TCP,80:31624/TCP,443:30723/TCP   16h
  4. Ensure all the runtime traffic is working by monitoring your dashboards. Only proceed to the next step if everything is working as expected. Make sure no traffic is going through your old ingress gateway (Anthos Service Mesh), as the DNS update may be slow to propagate because of DNS caching.
  5. To stop Apigee from supplying configuration to Anthos Service Mesh, follow the steps inStop supplying configuration to ASM in the Managing Apigee ingress gateway guide.
  6. Retest and monitor API proxy traffic.
  7. Follow the instructions in the Anthos Service Mesh documentation toUninstall Anthos Service Mesh from the cluster.

Upgrade Anthos Service Mesh to version 1.15

Perform this step only if you intend to continue using Anthos Service Mesh for your ingress gateway instead of usingApigee ingress gateway.You may experience a brief interruption of incoming traffic (typically less than two seconds) when you upgrade Anthos Service Mesh. If you are running hybrid in two or more clusters, you can avoid this interruption by upgrading one cluster at a time. Divert all production traffic away from the cluster you are about to upgrade, take the cluster offline, and then proceed with the upgrade process. Repeat the process for each cluster.

Perform the procedures using the Anthos Service Mesh documentation appropriate for your platform:

The instructions to install and configure Anthos Service Mesh are different depending on your platform. The platforms are divided into the following categories:

  • GKE: Google Kubernetes Engine clusters running on Google Cloud.
  • Outside Google Cloud: Anthos clusters running on:
    • Anthos clusters on VMware (GKE on-prem)
    • Anthos on bare metal
    • Anthos clusters on AWS
    • Amazon EKS
  • Other Kubernetes Platforms: Conformant clusters created and running on:
    • AKS
    • EKS
    • OpenShift

GKE

The sequence for upgrading to Anthos Service Mesh version 1.17.8 for your hybrid installation is as follows:

  1. Prepare for the upgrade.
  2. Install the new version of Anthos Service Mesh.
  3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
  4. Upgrade your gateways and configure the new webhooks.

Prepare to upgrade Anthos Service Mesh to version 1.17.8

Note: You can upgrade directly from Anthos Service Mesh version 1.11 or higher to Anthos Service Mesh version 1.17.8. If you are running a version of Anthos Service Mesh older than 1.11, you must first upgrade to version 1.11. Follow the instructions inUpgrading from earlier versions
  1. Review the requirements inUpgrade Anthos Service Mesh, but do not perform the upgrade yet.
  2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the currentistiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')echo $DELETE_REV

    Your output should look something like1.16

    Note: If the command returns multiple values, set the value ofDELETE_REV to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.
  3. Create a newoverlay.yaml file or verify that your existingoverlay.yaml contains the following contents:
    apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:components:ingressGateways:-name:istio-ingressgatewayenabled:truek8s:nodeSelector:# default node selector, if different or not using node selectors, change accordingly.cloud.google.com/gke-nodepool:apigee-runtimeresources:requests:cpu:1000mservice:type:LoadBalancerloadBalancerIP:STATIC_IP# If you do not have a reserved static IP, leave this out.ports:-name:http-status-portport:15021-name:http2port:80targetPort:8080-name:httpsport:443targetPort:8443meshConfig:accessLogFormat:'{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:Important: Make sure to follow the instructions to upgrade Anthos Service Mesh with optional features, and to include youroverlay.yaml.
    1. Download asmcli
    2. Grant cluster admin permissions
    3. Validate project and cluster
    4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section".
  5. Switch to the new control plane:
    1. Get the revision label that is onistiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27sasm-1178-1    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27sasm-1178-1
    2. Assign the newer revision label to an environment variable.

      In the output, under theREV column, note the value of the revision label for the new version. In this example, the value isasm-1178-1

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label toistio-system namespace and remove theistio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see"istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have theistio-injection label. Because auto-injection fails if a namespace has both theistio-injection and the revision label, allkubectl label commands in the Anthos Service Mesh documentation include removing theistio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  6. Delete the previous versions:
    1. Navigate into the directory where you installedasmcli.
    2. Store the output directory for your Anthos Service Mesh installation in theDIR_PATH environment variable. This is the same directory you specified in theUpgrade with optional features procedure.
      export DIR_PATH=OUTPUT_DIR
    3. Create a shell script containing the following commands:
      #!/bin/bashset -exif [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then  kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=truefi
    4. Execute the script to delete the previous versions.

Outside Google Cloud

These instructions cover upgrading Anthos Service Mesh on:

  • Anthos clusters on VMware (GKE on-prem)
  • Anthos on bare metal
  • Anthos clusters on AWS
  • Amazon EKS

The sequence for upgrading to Anthos Service Mesh version 1.17.8 for your hybrid installation is as follows:

  1. Prepare for the upgrade.
  2. Install the new version of Anthos Service Mesh.
  3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
  4. Upgrade your gateways and configure the new webhooks.

Prepare to upgrade Anthos Service Mesh to version 1.17.8

Note: You can upgrade directly from Anthos Service Mesh version 1.11 or higher to Anthos Service Mesh version 1.17.8. If you are running a version of Anthos Service Mesh older than 1.11, you must first upgrade to version 1.11. Follow the instructions inUpgrading from earlier versions
  1. Review the requirements inUpgrade Anthos Service Mesh, but do not perform the upgrade yet.
  2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the currentistiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')echo $DELETE_REV

    Your output should look something like1.16

    Note: If the command returns multiple values, set the value ofDELETE_REV to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.
  3. Create a newoverlay.yaml file or verify that your existingoverlay.yaml contains the following contents:
    apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:components:ingressGateways:-name:istio-ingressgatewayenabled:truek8s:nodeSelector:# default node selector, if different or not using node selectors, change accordingly.cloud.google.com/gke-nodepool:apigee-runtimeresources:requests:cpu:1000mservice:type:LoadBalancerloadBalancerIP:STATIC_IP# If you do not have a reserved static IP, leave this out.ports:-name:http-status-portport:15021-name:http2port:80targetPort:8080-name:httpsport:443targetPort:8443values:gateways:istio-ingressgateway:runAsRoot:truemeshConfig:accessLogFormat:'{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:Important: Make sure to follow the instructions to upgrade Anthos Service Mesh with optional features, and to include youroverlay.yaml.
    1. Download asmcli
    2. Grant cluster admin permissions
    3. Validate project and cluster
    4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section".
  5. Switch to the new control plane:
    1. Get the revision label that is onistiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27sasm-1178-1    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27sasm-1178-1
    2. Assign the newer revision label to an environment variable.

      In the output, under theREV column, note the value of the revision label for the new version. In this example, the value isasm-1178-1

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label toistio-system namespace and remove theistio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see"istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have theistio-injection label. Because auto-injection fails if a namespace has both theistio-injection and the revision label, allkubectl label commands in the Anthos Service Mesh documentation include removing theistio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  6. Delete the previous versions:
    1. Navigate into the directory where you installedasmcli.
    2. Store the output directory for your Anthos Service Mesh installation in theDIR_PATH environment variable. This is the same directory you specified in theUpgrade with optional features procedure.
      export DIR_PATH=OUTPUT_DIR
    3. Create a shell script containing the following commands:
      #!/bin/bashset -exif [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then  kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=truefi
    4. Execute the script to delete the previous versions.

AKS / EKS

In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version 1.17.8-asm.4-distroless on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

  1. Before installing the new version, determine the current revision. You will need this information to delete thevalidating webhook andmutating webhook from your current Anthos Service Mesh installation. Use the following command to store the currentistiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')echo $DELETE_REV

    Your output should look something like1.16

    Note: If the command returns multiple values, set the value ofDELETE_REV to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

    2. Linux

  2. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz
  3. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig
    openssldgst-verify/dev/stdin-signature1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig1.17.8-asm.4-distroless.tar.gz <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  4. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests/profiles directory.
  5. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  6. For convenience, add the tools in the/bin directory to yourPATH:
    export PATH=$PWD/bin:$PATH

    7. Mac OS

  7. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz
  8. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz.1.sig
    openssldgst-sha256-verify/dev/stdin-signature1.17.8-asm.4-distroless-osx.tar.gz.1.sig1.17.8-asm.4-distroless.tar.gz <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  9. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-osx.tar.gz

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests/profiles directory.
  10. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  11. For convenience, add the tools in the/bin directory to yourPATH:
    export PATH=$PWD/bin:$PATH

    12. Windows

  12. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip
  13. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip.1.sig
    openssldgst-verify--signature1.17.8-asm.4-distroless-win.zip.1.sig1.17.8-asm.4-distroless.win.zip <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  14. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-win.zip

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests\profiles directory.
  15. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  16. For convenience, add the tools in the \bin directory to your PATH:
    set PATH=%CD%\bin:%PATH%
  17. Now that Anthos Service Mesh Istio is installed, check the version ofistioctl:
    istioctl version
  18. Create a namespace called istio-system for the control plane components:
    kubectl create namespace istio-system

Installing Anthos Service Mesh

  1. Edit youroverlay.yaml file or create a new one with the following contents:
    apiVersion: install.istio.io/v1alpha1kind: IstioOperatorspec:  meshConfig:    accessLogFile: /dev/stdout    enableTracing: true    accessLogFormat:      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'  components:    ingressGateways:    - name: istio-ingressgateway      enabled: true      k8s:        service:          type: LoadBalancer          ports:          - name: status-port            port: 15021            targetPort: 15021          - name: http2            port: 80            targetPort: 8080          - name: https            port: 443            targetPort: 8443
  2. Install Anthos Service Mesh withistioctl using theasm-multicloud profile:
    istioctl install \    --set profile=asm-multicloud \    --set revision="asm-1178-1" \    --filename overlay.yaml

    Your output should look something like:

    kubectl get pods -n istio-systemNAME                                   READY   STATUS    RESTARTS   AGEistio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13sistio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57sistiod-asm-1178-1-798ffb964-2ls88       1/1     Running   0          3m21sistiod-asm-1178-1-798ffb964-fnj8c       1/1     Running   1          3m21s

    The--set revision argument adds a revision label in the formatistio.io/rev=asm-1178-1 toistiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particularistiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label onistiod.

  3. Verify that your install completed:
    kubectl get svc -n istio-system

    Your output should look something like:

    NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGEistio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35sistiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46sistiod-asm-1178-1       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
  4. Switch to the new control plane:
    1. Get the revision label that is onistiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27sasm-1178-1    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27sasm-1178-1
    2. Assign the newer revision label to an environment variable.

      In the output, under theREV column, note the value of the revision label for the new version. In this example, the value isasm-1178-1

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label toistio-system namespace and remove theistio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see"istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have theistio-injection label. Because auto-injection fails if a namespace has both theistio-injection and the revision label, allkubectl label commands in the Anthos Service Mesh documentation include removing theistio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  5. Delete the previous versions:
    1. Navigate into the directory where you installedasmcli.
    2. Create a shell script containing the following commands:
      #!/bin/bashset -exif [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=truefi
    3. Execute the script to delete the previous versions.

OpenShift

In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version 1.17.8-asm.4-distroless on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

  1. Before installing the new version, determine the current revision. You will need this information to delete thevalidating webhook andmutating webhook from your current Anthos Service Mesh installation. Use the following command to store the currentistiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')echo $DELETE_REV

    Your output should look something like1.16

    Note: If the command returns multiple values, set the value ofDELETE_REV to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

    2. Linux

  2. Grant theanyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  3. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz
  4. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig
    openssldgst-verify/dev/stdin-signature1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig1.17.8-asm.4-distroless.tar.gz <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  5. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests/profiles directory.
  6. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  7. For convenience, add the tools in the/bin directory to yourPATH:
    export PATH=$PWD/bin:$PATH

    8. Mac OS

  8. Grant theanyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  9. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz
  10. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz.1.sig
    openssldgst-sha256-verify/dev/stdin-signature1.17.8-asm.4-distroless-osx.tar.gz.1.sig1.17.8-asm.4-distroless.tar.gz <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  11. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-osx.tar.gz

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests/profiles directory.
  12. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  13. For convenience, add the tools in the/bin directory to yourPATH:
    export PATH=$PWD/bin:$PATH

    14. Windows

  14. Grant theanyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  15. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip
  16. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip.1.sig
    openssldgst-verify--signature1.17.8-asm.4-distroless-win.zip.1.sig1.17.8-asm.4-distroless.win.zip <<'EOF'-----BEGINPUBLICKEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZwQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==-----ENDPUBLICKEY-----EOF
  17. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf 1.17.8-asm.4-distroless-win.zip

    The command creates an installation directory in your current working directory named1.17.8-asm.4-distroless that contains:

    • Sample applications in thesamples directory.
    • Theistioctl command-line tool that you use to install Anthos Service Mesh is in thebin directory.
    • The Anthos Service Mesh configuration profiles are in themanifests\profiles directory.
  18. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd 1.17.8-asm.4-distroless
  19. For convenience, add the tools in the \bin directory to your PATH:
    set PATH=%CD%\bin:%PATH%
  20. Now that Anthos Service Mesh Istio is installed, check the version ofistioctl:
    istioctl version
  21. Create a namespace called istio-system for the control plane components:
    kubectl create namespace istio-system

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label onistiod. You need to set the same revision on the validating webhook.

  1. Create a file calledistiod-service.yaml with the following contents:
    apiVersion:v1kind:Servicemetadata:name:istiodnamespace:istio-systemlabels:istio.io/rev:asm-1178-1app:istiodistio:pilotrelease:istiospec:ports:-port:15010name:grpc-xds#plaintextprotocol:TCP-port:15012name:https-dns#mTLSwithk8s-signedcertprotocol:TCP-port:443name:https-webhook#validationandinjectiontargetPort:15017protocol:TCP-port:15014name:http-monitoring#prometheusstatsprotocol:TCPselector:app:istiodistio.io/rev:asm-1178-1meshConfig:accessLogFormat:'{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  2. Usekubectl to apply the validating webhook configuration:
    kubectl apply -f istiod-service.yaml
  3. Verify that the configuration was applied:
    kubectl get svc -n istio-system

    The response should look similar to:

    NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGEistiod   ClusterIP   172.200.18.133   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   22s

Installing Anthos Service Mesh

  1. Edit youroverlay.yaml file or create a new one with the following contents:
    apiVersion: install.istio.io/v1alpha1kind: IstioOperatorspec:  meshConfig:    accessLogFile: /dev/stdout    enableTracing: true    accessLogFormat:      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'  components:    ingressGateways:      - name: istio-ingressgateway        enabled: true        k8s:          service:            type: LoadBalancer            ports:            - name: status-port              port: 15021              targetPort: 15021            - name: http2              port: 80              targetPort: 8080            - name: https              port: 443              targetPort: 8443
  2. Install Anthos Service Mesh withistioctl using theasm-multicloud profile:
    istioctl install \    --set profile=asm-multicloud \    --set revision="asm-1178-1" \    --filename overlayfile.yaml

    Your output should look something like:

    kubectl get pods -n istio-systemNAME                                   READY   STATUS    RESTARTS   AGEistio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13sistio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57sistiod-asm-1178-1-798ffb964-2ls88       1/1     Running   0          3m21sistiod-asm-1178-1-798ffb964-fnj8c       1/1     Running   1          3m21s

    The--set revision argument adds a revision label in the formatistio.io/rev=1.6.11-asm.1 toistiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particularistiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label onistiod.

  3. Verify that your install completed:
    kubectl get svc -n istio-system

    Your output should look something like:

    NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGEistio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35sistiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46sistiod-asm-1178-1       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
  4. Switch to the new control plane:
    1. Get the revision label that is onistiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm    istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27sasm-1178-1    istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27sasm-1178-1
    2. Assign the newer revision label to an environment variable.

      In the output, under theREV column, note the value of the revision label for the new version. In this example, the value isasm-1178-1

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label toistio-system namespace and remove theistio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see"istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have theistio-injection label. Because auto-injection fails if a namespace has both theistio-injection and the revision label, allkubectl label commands in the Anthos Service Mesh documentation include removing theistio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  5. Delete the previous versions:
    1. Navigate into the directory where you installedasmcli.
    2. Create a shell script containing the following commands:
      #!/bin/bashset -exif [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=truefi
    3. Execute the script to delete the previous versions.
Congratulations! You have successfully upgraded to Apigee hybrid version 1.8.8.

Rolling back an upgrade

Follow these steps to roll back a previous upgrade:

  1. Clean up completed jobs for the hybrid runtime namespace, whereNAMESPACE is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace isapigee:
    kubectl delete job -nNAMESPACE \  $(kubectl get job -nNAMESPACE \  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  2. Clean up completed jobs for theapigee-system namespace:
    kubectl delete job -n apigee-system \  $(kubectl get job -n apigee-system \  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  3. Change theAPIGEECTL_HOME variable to point to the directory that contains the previous version ofapigeectl. For example:
    export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
  4. Undo the changes to youroverrides file:
    1. Remove or comment outingressGateways and all its properties.
    2. Set the value ofvirtualhosts.selector.app to the previous value, for example:
      virtualhosts:  - name: my-env-group    selector:      app: istio-ingressgateway
    3. Remove or comment outao.args.disableIstioConfigInAPIServer.
  5. In the root directory of the installation you want to roll back to, runapigeectl apply, check the status of your pods, and then runapigeectl init. Be sure to use the original overrides file for the version you wish to roll back to:
    1. In the hybrid-files directory, runapigeectl apply:
      $APIGEECTL_HOME/apigeectl apply -fORIGINAL_OVERRIDES_FILE

      WhereORIGINAL_OVERRIDES_FILE is relative path and filename of the overrides file for your previous version hybrid installation, for example,./overrides/overrides1.7.yaml.

    2. Check the status of your pods:
      kubectl -nNAMESPACE get pods

      WhereNAMESPACE is your Apigee hybrid namespace.

    3. Check the status ofapigeeds:
      kubectl describe apigeeds -n apigee

      Your output should look something like:

      Status:CassandraDataReplication:CassandraPodIps:10.8.2.204CassandraReadyReplicas:1Components:Cassandra:LastSuccessfullyReleasedVersion:Revision:v1-f8aa9a82b9f69613Version:v1Replicas:Available:1Ready:1Total:1Updated:1State:runningScaling:InProgress:falseOperation:RequestedReplicas:0State:running

      Proceed to the next step only when theapigeeds pod is running.

    4. Run the following command to make note of what your new replica count values will be for message processor after the upgrade.If these values do not match what you have set previously, change the values in your overrides file to match your previous configuration.
      apigeectl apply -fORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2

      Your output should look something like:

            autoScaler:        minReplicas: 2        maxReplicas: 10
    5. If you are rolling back to hybrid v1.8.4 or earlier, delete the controller deployment used by hybrid v1.8.5 and newer:
      kubectl -n apigee-system delete deploy apigee-controller-manager
    6. Runapigeectl init:
      $APIGEECTL_HOME/apigeectl init -fORIGINAL_OVERRIDES_FILE
  6. Delete the Apigee ingress gateway manager deployment. This component is relevant only to Apigee hybrid versions 1.8 and newer.
    kubectl delete deployment -nNAMESPACE apigee-ingress-gateway-manager

    WhereNAMESPACE is your Apigee hybrid namespace.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-19 UTC.