Cloud Service Mesh security best practices

Note: This guide only supports Cloud Service Mesh with Istio APIs and doesnot support Google Cloud APIs. For more information see,Cloud Service Mesh overview.

This document describes best practices to establish and govern a secureCloud Service Meshconfiguration running on Google Kubernetes Engine (GKE). The guidance in thedocument goes beyond the settings used to configure and installCloud Service Meshand describes how you can use Cloud Service Mesh with other Google Cloudproducts and features to protect against the security threats that applicationsin a mesh may face.

The intended audience for this document includes administrators who managepolicies in a Cloud Service Mesh and users who run services in aCloud Service Mesh. The security measures described here are also useful fororganizations that need to enhance the security of their service meshes to meetcompliance requirements.

The document is organized as follows:

• Introduction
• Attack vectors and security risks
  • Attack vectors
  • Security risks
• Measures to protect a service mesh
  • Security architecture
  • Cluster security
    • Enable strict mutual TLS
    • Enable access controls
      • Enforce Cloud Service Mesh authentication policies
      • JSON Web Token (JWT)
      • Cloud Service Mesh user authentication
      • Enforce authorization policies
      • Enforce token exchange for accessing mesh services
    • Securely handle Cloud Service Mesh policy exceptions
      • Enforce Kubernetes Network Policies
    • Secure control plane access
    • Enforce namespace boundaries
      • Enforce Kubernetes RBAC policies
  • Mesh edge security
    • Cluster ingress access control
    • Regulate cluster egress traffic
    • Use private cluster or VPC Service Control to lock down external accesses
    • Defend against external DDoS attacks
  • Security for mesh administration and automation
    • Segment the roles used for mesh operations
    • Automatically validate policy configurations
      • Use a GitOps approach with Config Sync to prevent configuration drift
    • Enforce Audit Logging and monitoring
    • Protect the certificate authority for in-cluster certificates
  • Workload security
    • Restrict Pod privileges
    • Secure container images
    • Mitigate against mesh vulnerabilities
    • Use Workload Identity Federation for GKE to securely access Google services
    • Monitor security status through security dashboard and telemetry
  • Security for sensitive user data and credentials

Introduction

Cloud Service Mesh provides features and tools that help you observe, manage, andsecure services in a unified way. It takes an application-centric approach anduses trusted application identities rather than a network IP-focused approach.You can deploy a service mesh transparently without the need to modify existingapplication code. Cloud Service Mesh provides declarative control over networkbehavior, which helps to decouple the work of teams that are responsible fordelivering and releasing application features from the responsibilities ofadministrators responsible for security and networking.

Cloud Service Mesh is based on the open sourceIstio service mesh, whichenables sophisticated configurations and topologies. Depending on the structureof your organization, one or more teams or roles may be responsible forinstalling and configuring a mesh. The default Cloud Service Mesh settings are chosento protect applications, but in some cases, you may needcustom configurations or to grant exceptions by excluding certain apps, ports,or IP addresses from participating in a mesh. Having controls in place to governmesh configurations and security exceptions is important.

Note: This guide complementsIstio's security best practices documentation,which includes detailed configuration recommendations for mutual TLS (mTLS),authorization policies, gateways, and other security configurations. You shouldtreat these recommendations as a foundation to be used together with the bestpractices discussed in this guide. This guide describes additional bestpractices for Cloud Service Mesh and how technologies inGoogle Cloud can secure all layers, components, and information flowsin a mesh.

Attack vectors and security risks

Attack vectors

Cloud Service Mesh security follows the zero trust security model whichassumes security threats originate from both inside and outside of anorganization's security perimeter. Examples of security attack types that maythreaten applications in a service mesh include:

• Data exfiltration attacks. For example, attacks that eavesdrop on sensitivedata or credentials from service-to-service traffic.
• Man-in-the-middle attacks. For example, a malicious service that masqueradesas a legitimate service to obtain or modify the communication betweenservices.
• Privilege escalation attacks. For example, attacks that use illicit accessto elevated privileges to conduct operations in a network.
• Denial of service (DoS) attacks.
• Botnet attacks that try to compromise and manipulate services to launchattacks on other services.

The attacks can also be categorized based on the attack targets:

• Mesh internal network attacks. Attacks aimed at tampering, eavesdropping, orspoofing the mesh internal service-to-service or service-to-control-planecommunication.
• Control plane attacks. Attacks aimed at causing the control plane tomalfunction (such as a DoS attack), or exfiltrating sensitive data from thecontrol plane.
• Mesh edge attacks. Attacks aimed at tampering, eavesdropping, or spoofingthe communication at the mesh ingress or egress.
• Mesh operation attacks. Attacks aimed at the mesh operations. Attackers maytry to obtain elevated privileges to conduct malicious operations in a mesh,such as modifying its security policies and workload images.

Security risks

Besides security attacks, a mesh also faces other security risks. The followinglist describes a few possible security risks:

• Incomplete security protection. A service mesh has not been configured withauthentication and authorization policies to protect its security. Forexample, no authentication or authorization policies are defined forservices in a mesh.
• Security policy exceptions. To accommodate their specific use cases, usersmay create security policy exceptions for certain traffic (internal orexternal) to be excluded from Cloud Service Mesh security policies. Tosecurely handle such cases, please refer to the sectionSecurely handle exceptions to policies.
• Neglect of image upgrades. Vulnerabilities may be discovered for the imagesused in a mesh. You need to keep the mesh component and workload imagesup-to-date with the latest vulnerability fixes.
• Lack of maintenance (no expertise or resources). The mesh software andpolicy configurations need regular maintenance to take advantage of thelatest security protection mechanisms.
• Lack of visibility. Misconfiguration or insecure configurations of meshpolicies and abnormal mesh traffic/operations are not brought to theattention of mesh administrators.
• Configuration drift.The configuration of policies in a mesh deviates from the source of truth.

Measures to protect a service mesh

This section presents an operating manual to secure service meshes.

Security architecture

The security of a service mesh depends on the security of the components atdifferent layers of the mesh system and its applications. The high-levelintention of the proposed Cloud Service Mesh security posture is to secure aservice mesh through integrating multiple security mechanisms at differentlayers, which jointly achieve the overall system security under the zero-trustsecurity model. The following diagram shows the proposed Cloud Service Meshsecurity posture.

Cloud Service Mesh provides security at multiple layers, including:

• Mesh edge security
  • Cloud Service Mesh ingress security provides access control forexternal traffic and secures external access to the APIs exposed by theservices in the mesh.
  • Cloud Service Mesh egress security regulates the outbound trafficfrom internal workloads.
  • Cloud Service Mesh User Authintegrates with Google infrastructure to authenticate external callsfrom web browsers to the services that run web applications.
  • Cloud Service Meshgateway certificate managementprotects and rotates the private keys and X.509 certificates used byCloud Service Mesh ingress and egress gateways usingCertificate Authority Service.
  • Cloud Armor can defend againstexternal distributed denial-of-service (DDoS) and Layer 7 attacks. Itserves as a Web Application Firewall (WAF) to protect the mesh fromnetwork attacks. For example, injection and remote code executionattacks.
  • VPC and VPC Service Controlsprotect the mesh edge through the private network access controls.
• Cluster security
  • Cloud Service Mesh mutual TLS (mTLS) enforces workload-to-workloadtraffic encryption and authentication.
  • Managed CA, such as Cloud Service Mesh certificate authority andCertificate Authority Service, securely provisions and manages certificates usedby the workloads.
  • Cloud Service Mesh authorization enforces access control for meshservices based on their identities and other attributes.
  • GKE Enterprise security dashboardprovides monitoring of the configurations of security policies andKubernetes Network Policies for the workloads.
  • Kubernetes Network Policy enforces Pod access control based on IPaddresses, Pod labels, namespaces, and more.
  • Control plane security defends against attacks on the control plane.This protection prevents attackers from modifying, exploiting, orleaking service and mesh configuration data.
• Workload security
  • Stay up-to-date with Cloud Service Mesh security releases to ensurethe Cloud Service Mesh binaries running in your mesh are free ofpublicly known vulnerabilities.
  • Workload Identity Federation for GKEenables workloads to obtain credentials to securely call Google services.
  • Kubernetes CNI (Container Network Interface) prevents privilegeescalation attacks by eliminating the need for a privilegedCloud Service Mesh init container.
• Operator security
  • Kubernetes role-based access control (RBAC) restricts access toKubernetes resources and confines operator permissions to mitigateattacks originating from malicious operators or operator impersonation.
  • GKE Enterprise Policy Controllervalidates and audits policy configurations in the mesh to preventmisconfigurations.
  • Google Cloud Binary Authorizationensures that the workload images in the mesh are the ones authorized bythe administrators.
  • Google Cloud Audit Logging audits mesh operations.

The diagram below shows the communication and configuration flows with theintegrated security solutions in Cloud Service Mesh.

Cluster security

Enable strict mutual TLS

A man-in-the-middle (MitM) attack tries to insert a malicious entity between twocommunicating parties in order to eavesdrop on or manipulate the communication.Cloud Service Mesh defends against MitM and data exfiltration attacks byenforcingmTLS authentication and encryptionfor all communicating parties. Permissive mode uses mTLS when both sides supportit, but allows connections without mTLS. By contrast, strict mTLS requires thattraffic be encrypted and authenticated with mTLS and does not allow plain texttraffic.

Cloud Service Mesh allows you toconfigure the minimum TLS versionfor the TLS connections among your workloads to meet your security andcompliance requirements.

For more information, seeCloud Service Mesh by example: mTLS | Enforcing mesh-wide mTLS.

Enable access controls

Cloud Service Mesh security policies (such as authentication and authorizationpolicies) should be enforced on all traffic in and out of the mesh unless thereare strong justifications to exclude a service or Pod from Cloud Service Meshsecurity policies. In some cases, users may have legitimate reasons to bypassCloud Service Mesh security policies for some ports and IP ranges. Forexample, to establish native connections with services not managed byCloud Service Mesh. To secure Cloud Service Mesh under such use cases,please refer toSecurely handle Cloud Service Mesh policy exceptions.

Service access control is critical in preventing unauthorized access toservices. mTLS enforcement encrypts and authenticates a request but a mesh stillneedsCloud Service Mesh authorization policiesto enforce access control on services. For example, rejecting an unauthorizedrequest coming from an authenticated client.

Cloud Service Mesh authorization policies provide a flexible way to configureaccess controls to defend your services against unauthorized access.Cloud Service Mesh authorization policies should be enforced based on theauthenticated identities derived from the authentication results - mTLS or JSONWeb Token (JWT) based authentications should be used together as part ofCloud Service Mesh authorization policies.

Enforce Cloud Service Mesh authentication policies

JSON Web Token (JWT)

In addition to mTLS authentication, mesh administrators can require a service toauthenticate and authorize requests based on JWT.Cloud Service Mesh does not act as a JWT provider but authenticates JWTsbased on the configured JSON web key set (JWKS) endpoints. JWT authenticationcan be applied to ingress gateways for external traffic or to internal servicesfor in-mesh traffic. JWT authentication can be combined with mTLS authenticationwhen a JWT is used as a credential to represent the end caller and the requestedservice requires proof that it is being called on behalf of the end caller.Enforcing JWT authentication defends against attacks that access a servicewithout valid credentials and on behalf of a real end user.

Cloud Service Mesh user authentication

Cloud Service Mesh user authenticationis an integrated solution for browser-based end-user authentication and accesscontrol to your workloads. It integrates a service mesh with existing IdentityProviders (IdP) to implement a standard web-based OpenID Connect (OIDC) loginand consent flow and uses Cloud Service Mesh authorization policies for accesscontrol.

Enforce authorization policies

Cloud Service Mesh authorization policies control:

• Who or what is allowed to access a service.
• Which resources can be accessed.
• Which operations can be conducted on the allowed resources.

Authorization policies are a versatile way to configure access control based onthe actual identities that services run as, application layer (Layer 7)properties of traffic (for example request headers), and network layer (Layer 3and Layer 4) properties like IP ranges and ports.

Cloud Service Mesh authorization policies should be enforced based on authenticatedidentities derived from the authentication results to defend againstunauthorized access to services or data.

By default, access to a service should be denied unless an authorization policyis explicitly defined to allow access to the service. SeeAuthorization Policy Best Practicesfor examples of authorization policies that deny access requests.

Authorization policies should restrict trust as much as possible. For example,the access to a service can be defined based on individual URL paths exposed bya service such that only a service A can access the path/admin of a service B.

Authorization policies can be used together withKubernetes Network Policies,which only operate at the network layer (Layer 3 and Layer 4) and control thenetwork access for IP addresses and ports on Kubernetes Pods and Kubernetesnamespaces.

Note: Cloud Service Mesh does not handle non-TCP traffic (for example UDP), so itcannot be controlled by authorization policies. Mesh administrators can useKubernetes Network Policies to define policies for UDP traffic. For example, ifmesh services do not use UDP then mesh administrators can define a Kubernetesnetwork policy to block UDP traffic.

Enforce token exchange for accessing mesh services

To defend against token replay attacks which steal tokens and re-use the stolentokens to access mesh services, a token in a request from outside the meshshould be exchanged for a short-lived mesh-internal token at the mesh edge.

A request from outside the mesh to access a mesh service needs to include atoken, such as JWT or cookie, in order to be authenticated and authorized by themesh service. A token from outside the mesh may be long-lived. To defend againsttoken replay attacks, a token from outside the mesh should be exchanged for ashort-lived mesh-internal token with a limited scope at the ingress of the mesh.The mesh service authenticates a mesh-internal token and authorizes the accessrequest based on the mesh-internal token.

What's next

• Review best practices for using Cloud Service Mesh egress gateways on GKE clusters
• Configure transport security
• Update your authorization policies