Enable optional features on an in-cluster control plane

This page describes how to enable optional features on Cloud Service Meshwith an in-cluster control plane.

When you install in-cluster Cloud Service Mesh, thefeatures that are enabled by default differ by platform.You can override the default configuration and enable an optional feature byincluding anoverlay file when you install (or upgrade) Cloud Service Mesh. Anoverlay file is a YAML file containing anIstioOperator custom resource (CR)that you use to configure the control plane. Specify one feature per overlay file. You can layer on more overlays,and each overlay file overrides the configuration on the previous layers.

The overlay files on this page are in theanthos-service-mesh package in GitHub. These files contain common customizations to the defaultconfiguration. You can use these files as they are, or you can make additionalchanges to them as needed.

When you install Cloud Service Mesh usingasmcli script, youcan specify one or more overlay files with the--option or the--custom_overlay options. If you don't need to make any changes to thefiles in theanthos-service-mesh repository, you can use--option, andthe script fetches the file from GitHub for you. Otherwise, you can makechanges to the overlay file, and then use the--custom_overlay option topass it to theasmcli.

Important: Don't specify more than oneIstioOperator CR in an overlay filebecauseistioctl install only applies the first CR that it finds in the file.

Don't include multiple CRs in one overlay file	Create separate overlay files for each CR

How to enable optional features

The following examples are simplified to show only using the custom overlays toenable optional features. ReplaceOTHER_FLAGS with therequired installation flags.

• Use--option whenyou don't need to make any changes to the overlay file. With--option,asmcli fetches the file from the GitHub repository for you, so you musthave an internet connection.

  ./asmcliinstall\OTHER_FLAGS\--optionOPTION_NAME

  ReplaceOPTION_NAME with the option you want to enable.Be sure to omit the .yaml extension and only include the name of the overlayfile, such asiap-operator andattached-cluster.For a list of options refer to theanthos-service-mesh package.

• Use--custom_overlay whenyou need to customize the overlay file.

  ./asmcliinstall\OTHER_FLAGS\--custom_overlayPATH_TO_FILE

  ReplacePATH_TO_FILE with the path to the overlayfile you want to use.

YAML for optional features

The following sections provide the YAML to enable optional and supportedfeatures.

mTLSSTRICT mode

Theglobal.mtls.enabled configuration was removed from theIstioOperatorCR to avoid issues with upgrades and to provide a more flexible installation.To enableSTRICT mTLS,configure apeer authentication policy instead.

Distroless proxy image

As a best practice, you should restrict the contents of a container runtime to only the necessary packages. This approach improves security and the signal-to-noise ratio of Common Vulnerabilities and Exposures (CVE) scanners.Istio provides proxy images based ondistroless base images.

The following configuration enables distroless images for the entire Cloud Service Mesh.An image type change requires each pod to restart and get re-injected to take effect.

apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:defaultConfig:image:imageType:distroless

The distroless image does not contain any binaries other than the proxy. It is therefore not possible toexec a shell or usecurl,ping, or other debug utilities inside the container.

If you run a curl command, you see the following error:

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec  "<container-id>"OCI runtime exec failed: exec failed: unable to start container process: exec: "curl": executable file not found in $PATH: unknown

If you run a shell command, you see the following error:

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "<container-id>"OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown

If you need access to these tools for specific pods, you may override theimageType by using the following pod annotation.

sidecar.istio.io/proxyImageType: debug

After changing the image type of a deployment via the annotation, the deployment should be restarted.

kubectl rollout restart deployment -nNAMESPACEDEPLOYMENT_NAME

For most types of proxy debugging,istioctl proxy-cmd should be used which does not require a debug base image.

Use a custom overlay for custom registry

You can use a custom overlay for custom registries, such as if you need toinstall Cloud Service Mesh from a custom container registry. For example:

apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:hub:{private_registry_url}

The following is a list of images for Cloud Service Mesh that you need to mirror tothe custom container registry:

• Install-cni -gke.gcr.io/asm/install-cni:1.28.2-asm.4
• Managed Data Plane -gke.gcr.io/asm/mdp:1.28.2-asm.4
• Pilot -gke.gcr.io/asm/pilot:1.28.2-asm.4
• Proxyv2 -gke.gcr.io/asm/proxyv2:1.28.2-asm.4

Add images to a private registry

To push Cloud Service Mesh images to a private registry, complete the followingsteps.

1. Pull the Cloud Service Mesh images:

   docker pull gke.gcr.io/asm/install-cni:1.28.2-asm.4docker pull gke.gcr.io/asm/mdp:1.28.2-asm.4docker pull gke.gcr.io/asm/pilot:1.28.2-asm.4docker pull gke.gcr.io/asm/proxyv2:1.28.2-asm.4

2. Create a variable for your private registry URL:

   export PRIVATE_REGISTRY_URL=PRIVATE_REGISTRY_URL

   ReplacePRIVATE_REGISTRY_URL with your private registryURL.
3. Tag the images with your private registry URL:

   docker tag gke.gcr.io/asm/install-cni:1.28.2-asm.4 \ ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/install-cni:1.28.2-asm.4docker tag gke.gcr.io/asm/mdp:1.28.2-asm.4 \ ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/mdp:1.28.2-asm.4docker tag gke.gcr.io/asm/pilot:1.28.2-asm.4 \ ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/pilot:1.28.2-asm.4docker tag gke.gcr.io/asm/proxyv2:1.28.2-asm.4 \ ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/proxyv2:1.28.2-asm.4

4. Push the tagged images to your private registry:

   docker push ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/install-cni:1.28.2-asm.4docker push ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/mdp:1.28.2-asm.4docker push ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/pilot:1.28.2-asm.4docker push ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/proxyv2:1.28.2-asm.4

5. (Optional) If you use acanonical service, then add thecanonical service images to your private registry.
   1. Pull the Cloud Service Mesh canonical service images:

              docker pull gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1        docker pull gke.gcr.io/asm/canonical-service-controller:1.10.3-asm.16

   2. Tag the images with your private registry URL:

              docker tag gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 \        ${PRIVATE_REGISTRY_URL}/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1        docker tag gke.gcr.io/asm/canonical-service-controller:1.10.3-asm.16 \        ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/canonical-service-controller:1.10.3-asm.16

   3. Push the tagged images to your private registry:

              docker push ${PRIVATE_REGISTRY_URL}/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1        docker push ${PRIVATE_REGISTRY_URL}/gke.gcr.io/asm/canonical-service-controller:1.10.3-asm.16

If you can pull the tagged images from your private registry, then the procedurewas successful.

Increase termination drain duration

By default Envoy will wait five seconds (5s) for existing connections to complete when a pod is terminating.

PodterminationGracePeriodSeconds must be greater thanterminationDrainDuration value.

For more information seeGlobal Mesh Options.

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:defaultConfig:terminationDrainDuration:30s

Enable Access logs

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:accessLogFile:"/dev/stdout"

For more information, seeEnable Envoy's access logging.

Cloud Trace

Cloud Trace is available with Cloud Service Mesh installations on the followingplatforms:

• GKE on Google Cloud
• GKE Enterprise clusters on-premises if you install withCloud Service Mesh certificate authority

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:enableTracing:truevalues:global:proxy:tracer:stackdriver

For more information, seeAccessing traces.

Egress via egress gateways

We recommend that you install an injected gateway as described inInstall and upgrade gateways.Injection, or auto-injection, refers to the use ofmutating webhooksto modify Pod specifications at creation time. You use injection to add theEnvoy proxy sidecar configuration for your mesh services or to configure theEnvoy proxy of gateways.

Istio Container Network Interface

How you enable the Istio Container Network Interface (CNI) depends onthe environment that Cloud Service Mesh is installed on.

1. Enable a network policy.

2. Choose the overlay file that matches your platform.

Enable CNI on GKE

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:components:cni:enabled:truenamespace:kube-systemvalues:cni:cniBinDir:/home/kubernetes/binexcludeNamespaces:-istio-system-kube-system

Enable CNI on-premises

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:components:cni:enabled:truenamespace:kube-systemvalues:cni:cniBinDir:/opt/cni/binexcludeNamespaces:-istio-system-kube-system-gke-system

Enable Traffic logs for off-Google Cloud

Installing Cloud Service Mesh with Istio CA outside of Google Cloud reportsmetrics to Prometheus by default. Use this option to enable reporting Trafficlogs instead, or both Prometheus and Stackdriver, so you can use theCloud Service Mesh dashboards.

Only Stackdriver

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:values:telemetry:enabled:truev2:enabled:trueprometheus:enabled:falsestackdriver:enabled:true

Stackdriver and Prometheus

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:values:telemetry:enabled:truev2:enabled:trueprometheus:enabled:truestackdriver:enabled:true

Enable an internal load balancer

We recommend that you install aninjected gateway as described inInstall and upgrade gateways to set up an internalload balancer on GKE. When configuring the gateway Service,you include the annotation:networking.gke.io/load-balancer-type: "Internal"

External certificate management on the ingress gateway

For information on enabling external certificate management on the ingressgateway using Envoy SDS, seeSecure Gateways.