This document explains the security measures built into Connect.

An effective hybrid and multi-cloud platform delivers central management,observability, and access to services across environments. GKE Enterpriseprovides a uniform and comprehensive experience across those capabilities, nomatter what Kubernetes provider you use. Connect is alightweight agent that provides the following at economies of scale, and acrossproviders:

• Multi-cluster management and cluster visibility
• Application services deployment and lifecycle management

This document discusses the following:

• How the design of Connect puts security first
• Best practices for the Connect Agent deployment
• How to improve your Kubernetes deployment security posture

Architecture of Connect

Connect allows end users and Google Cloudservices to access Kubernetes API servers thataren't on the public internet. The Connect Agent runs inthe Kubernetes cluster (one agent per cluster), and connects to aConnect proxy. Google Cloud services that need tointeract with the Kubernetes cluster connect to the proxy, whichforwards requests to the agent. The agent, in turn, forwards them to theKubernetes API server as depicted in the following diagram.

When the agent is deployed, it establishes a persistent TLS 1.2+ connection toGoogle Cloud to wait for requests. Google Cloud services, when enabledby admins, can generate requests for your Kubernetes clusters. These requestsmight also come from direct user interaction with the Google Cloud console, Connect Gateway, or other Google Services.

The Google Cloud service sends the request to the proxy. The proxy thenforwards the request to the deployed agent responsible for a cluster, andfinally the agent forwards the request to the Kubernetes API server.The Kubernetes API server applies Kubernetes authentication,authorization, and audit-logging policies, and returns a response. The responseis passed back through the agent and the proxy to the Google Cloud service. At each step in the process, components perform per-connection and per-requestauthentication and authorization.

The Kubernetes API server applies the same authentication,authorization, and audit-logging policies to all requests, including requeststhrough Connect. This process ensures that you're alwaysin control of the access to your cluster.

Connect and defense-in-depth

Defense-in-depthis intrinsic to everything Google Cloud does within its infrastructure andsecurity practices. We take a layered approach to every aspect of securing ourorganization and our customers in order to protect valuable data, information,and users. This is the same principle by which we've designed Connect.

In addition to Google's own defense-in-depth strategy and design, you shouldevaluate the content provided here alongside your security posture andstandards. This section calls out additional measures that you cantake that complement Kubernetes hardening best practices.

Component-to-component security

Each component of a Connect request authenticates itspeers, and only shares data with peers that are both authenticated andauthorized for that data, as illustrated in the following diagram.

Each component of a Connect request uses the following toauthenticate each other:

• Transport Layer Security (TLS)
• Application Layer Transport Security (ALTS)
• OAuth

Each component of a Connect request uses the following toauthorize each other:

• Identity and Access Management (IAM)
• Allowlists

Each connection between the Kubernetes cluster and Google Cloud isencrypted, and at least one peer of each connection uses certificate-basedauthentication. This process helps to ensure that all token credentials areencrypted in transit, and only received by authenticated and authorized peers.

User authentication to Google Cloud

When using the Google Cloud console, users go through thestandard Google login flow.Google Cloud provides a TLS certificate that the user's browserauthenticates, and the user logs in to a Google Cloud or Google Workspace account toauthenticate to Google Cloud.

Access to a project through the Google Cloud console or other APIs iscontrolled by IAM permissions.

Google Cloud service-to-service authentication

Google Cloud uses ALTS for internal service-to-service authentication.ALTS allows Google Cloud services, such as the proxy, to create anauthenticated, integrity-protected connection.

Google Cloud services must be internally authorized to use the proxy toconnect to a remote Connect instance because the proxyuses aallowlist of service identities to limit access.

Authenticating Google Cloud

The agent uses TLS to authenticate and encrypt each connection. The agentauthenticates Google Cloud TLS certificates by using a set of rootcertificates built into the binary, to avoid inadvertently trusting certificatesadded to the agent's container. The agent only executes API calls againstcorrectly authenticated endpoints. This process helps to ensure that serviceaccount certificates and the Kubernetes API requests are sent byGoogle Cloud, and that any responses are sent only to Google Cloud.

For the list of domains that the agent communicates with during normal operation,seeEnsure network connectivity.

You can configure the agent toconnect to Google Cloud through an HTTP proxy. In this configuration, the agent uses theHTTP/1.1CONNECT against the HTTP proxy and establishes a TLS connection toGoogle Cloud. The HTTP proxy only sees the encrypted traffic between theagent and Google Cloud. The end-to-end integrity and security of connectionbetween the agent and Google Cloud is unaffected.

Authenticating the agent

The agent authenticates to Google Cloud by usingFleet Workload Identity Federation.Fleet Workload Identity Federation lets you authenticate from fleet workloads to Google Cloud APIs by using built-in Google Cloud and Kubernetes authentication mechanisms. When the agent wants to authenticate to Google Cloud, it exchanges its Kubernetes Service Account token for an access token that represents its workload identity.

Kubernetes API server

The agent uses the Kubernetesclient library to create a TLS connection to the Kubernetes API server. The Kubernetesruntime provides the agent's container with a TLS certificate authority (CA)certificate that the agent uses to authenticate the API server.

The API server authenticates each request separately, as described in the nextsection.

Request security

Each request sent from Google Cloud through Connectincludes credentials that identify the request's sender: both theGoogle Cloud service that generated the request, and (where applicable)the end user for whom the request is sent. These credentials allows theKubernetes API server to provide authorization and auditing controlsfor each request.

Service-to-agent authentication

Each request sent to the agent includes a short-lived token identifying theGoogle Cloud service that sent the request, as illustrated in the followingdiagram.

The token is signed by a Google Cloud service account associatedexclusively with the Google Cloud service. The agentfetches the service account's public keys to validate the token. This token isn't forwarded to the API server.

The agent validates Google Cloud certificates using CA roots embedded inthe binary. This process helps to ensure that it is receiving authentic andunaltered requests from Google Cloud.

End-user authentication

Google Cloud services that access clusters on behalf of a user requirethat user's credentials to authenticate to the API server, as illustrated in thefollowing diagram.

This policy helps to ensure that the same set of permissions are applied to theuser when accessing through Connect. SomeGoogle Cloud services authenticate to the API server on behalf of a user.For example, a user can access the Google Cloud console to view workloads inFleet-enrolled clusters. When a user accesses these services, they providecredentials that the Kubernetes API server recognizes:any of the tokens that the Kubernetes API server supports.

The Google Cloud console stores these credentials as part of a user's profile.These credentials are encrypted at rest, are only accessible with the user'sGoogle Cloud or Google Workspace credentials, and are only used forconnections through Connect. These credentials cannot bedownloaded again. The credentials are deleted when the user logs out of thecluster, when the cluster registration is deleted in Google Cloud, whenthe project is deleted, or when the user account is deleted. For moreinformation, seeData deletion on Google Cloud.

When a user interacts with the Google Cloud console, it generates requests forthe Kubernetes API server. The service sends the user's credentialsalong with the request through Connect. The agent thenpresents the request and credentials to the Kubernetes API server.

The Kubernetes API server authenticates the user's credentials,performs authorization on the user's identity, produces an audit event for theaction(if configured),and returns the result. Because the user-provided credentials are used toauthenticate the request, the Kubernetes API server applies the sameauthorization and auditing policy for Connectrequests as it does for other requests.

Service-to-Kubernetes authentication

Google Cloud services that access the Kubernetes API serveroutside of a user's context use Kubernetes impersonation to authenticate to theKubernetes API server. This method allows the Kubernetes APIserver to provide per-service authorization checks and audit logging, asillustrated in the following diagram.

Services at Google Cloud can use Connect outside ofa user's context. For example, a multicluster ingress service can automaticallysynchronize ingress resources across clusters. These services don't havecredentials that the Kubernetes API server can authenticate: most APIservers aren't configured to authenticate Google Cloud service'scredentials. However, an API server can delegate limited authenticationprivileges to another service throughimpersonation, and the agent can authenticate Google Cloud servicessending requests through Connect. Together, these allowrequests through the agent to authenticate as Google Cloud serviceaccounts.

When a Google Cloud service sends a request on its own behalf (rather thanin a user's context), the agent adds its ownKubernetes credentials,andKubernetes impersonation headers that identify the Google Cloud service, to the request. The impersonationheaders claim a user name of the Google Cloud service account authenticatedby the agent.

The Kubernetes API server authenticates the agent's credentials, andalso checks that the agentcan impersonate the Google Cloud service account. The ability to impersonate is typicallycontrolled by role-based access control (RBAC) rules, and can be limited tospecific identities, such as Google Cloud service accounts.

If the agent is authorized to impersonate the requested identity, the API serverthen performsauthorization checks for the Google Cloud service account, and serves the request. Theaudit log for the request includes both the agent's identity and theimpersonated Google Cloud service account.

In-cluster security

The agent ultimately sends Kubernetes API requests to theKubernetes API server, as illustrated in the following diagram.

The Kubernetes API server authenticates, authorizes, and audit-logsthese requests, just as it does for all other requests it serves.

As a proxy for these requests, the agent has access to sensitive data, such ascredentials, requests, and responses. Kubernetes and the Kubernetes ecosystemprovide a set of tools to prevent other actors from getting that access, and forhelping to ensure that the agent only accesses what it's supposed to.

Kubernetes authentication

The Kubernetes API server authenticates the sender of each incomingrequest to determine what permissions to apply in the authorization stage. Aspreviously described, the request either includes a user's credentials, orincludes the agent's Kubernetes credentials and impersonation headers.

Cluster admins remain in control of authentication mechanisms recognized by theKubernetes API server. Admins might be able to revoke a user'scredentials, and can revoke or reduce the privilege of the agent'scredentials.

Kubernetes authorization

The Kubernetes API server checks that the authenticated identity isallowed to take the requested action on the requested resource.

The cluster admin can use any of theKubernetes authorization mechanisms to configure authorization rules. Connect doesn't performany authorization checks on behalf of the cluster.

Agent security

The agent has access to its own (Kubernetes and Google Cloud) credentials,as well as the credentials, requests, and responses that pass through it. Assuch, the agent occupies a trusted position in a connected cluster.

The agent is designed with the following security fundamentals:

• The agent is written inGo,which provides garbage-collected memory management, and prevents manyunsafe memory operations.
• The agent is deployed in adistroless container image. The agent's image doesn't include ashell,libc,or other code that is extraneous to the agent's execution path.
• The agent's image is built by Google's shared build infrastructure fromchecked-in code. Only this build system can deploy agent images toContainer Registry.Google Cloud developers cannot deploy new images on their own. Thisprocess helps to ensure that all edits to the agent's source can be tracedback to an author and reviewer for non-repudiation.

The agent runs as a standarddeployment in a Kubernetes cluster that deploys at the time that you register your cluster.As a result, all of the options and best practices available for monitoring andsecuring deployments,ReplicaSets, and pods are available for the agent.

These mechanisms are designed to make it difficult to compromise the agentcontainer. However, privileged access to the agent's node can still compromisethe agent's environment; therefore, it is important for administrators to followstandard Kubernetes security guidelines for protecting cluster infrastructure.

Data security with VPC Service Controls

VPC Service Controls provides an additional layer of security defense forGoogle Cloud services that is independent of Identity and Access Management(IAM). While IAM enables granularidentity-basedaccess control, VPC Service Controls enables broadercontext-based perimetersecurity, including controlling data egress across the perimeter—forexample, you can specify that only certain projects can access yourBigQuery data. You can find more about how VPC Service Controls worksto protect your data in the VPC Service ControlsOverview.

You can use VPC Service Controls with Connect for extra data security,once you ensure that the necessary services to use Connect can beaccessed from within your specified service perimeter.

If you want to use VPC Service Controls, you need to enable the following APIs:

• cloudresourcemanager.googleapis.com
• gkeconnect.googleapis.com
• gkehub.googleapis.com

You also need to set upprivateconnectivity for access to therelevant APIs. You can find out how to do this inSet up privateconnectivity.