GKE Dataplane V2 observability provides GKE Dataplane V2 metrics and insightsinto workloads on GKE clusters on Google Cloud.With GKE Dataplane V2 observability, starting withGKE versions1.28 or later, you can:

• Capture, observe, and alert on network metrics usingGoogle Cloud Managed Service for Prometheus andCloud Monitoring withMetrics Explorer
• Understand traffic flows for a particular Service in a cluster
• Understand and identify issues with the network health of a Kubernetes workload
• VerifyKubernetes Network Policies

GKE Dataplane V2 observability offers the following troubleshooting tools:

• A Kubernetes cluster Network Topology
• A Kubernetes Network Policy verdict table with live traffic flows andconnection information
• Command-line tooling for troubleshooting Kubernetes traffic flows

GKE Dataplane V2 metrics

GKE Dataplane V2 metrics provide traffic flow information for the following:

• Traffic flows: insights about how GKE handles flowsbetween Pods and Services.

• Network policy enforcement: information about how GKEenforces Kubernetes Network Policies.

  Note: Observability draws insights from Layer 3 and Layer 4 network data.This aligns with GKE's Network Policies, which also functionat Layer 3 and Layer 4.

You can use GKE Dataplane V2 metrics to monitor and troubleshoot Kubernetesworkloads using the following tools:

• Google Cloud Managed Service for Prometheus to view and analyze yourGKE Dataplane V2 metrics. You can modify the Google Cloud Managed Service for Prometheusconfiguration to add or remove the metrics of your choice forGoogle Cloud Managed Service for Prometheus ingestion.
• Cloud Monitoring Metrics Explorer to view Pod-level traffic flowdetails.
• Cloud Monitoring to explore and use any metric. For example, you cancreate alerts that trigger when GKE Dataplane V2 metrics exceed certainthresholds.
• Self-managed Grafana to visualize metrics collected byGoogle Cloud Managed Service for Prometheus.

When you enable Google Cloud Managed Service for Prometheus:

• GKE creates aPodMonitoring resource
• GKE exposes the metrics endpoint

To consume metrics with Google Cloud Managed Service for Prometheus and to be able to create aPodMonitoring resource, you must enable Google Cloud Managed Service for Prometheuson the cluster. If you don't enable Google Cloud Managed Service for Prometheus, GKEexposes the metrics endpoint but does not create aPodMonitoringresource.

When you enable GKE Dataplane V2 metrics for a cluster,Google Cloud Managed Service for Prometheus ingests the following GKE Dataplane V2 metrics:

Enabling GKE Dataplane V2 metrics opens the metrics port on each Kubernetes node.

Additional metrics are also available, including from the open sourceobservability platformHubble.By default, Google Cloud Managed Service for Prometheus doesn't ingest these additional metricsbut you can configure Google Cloud Managed Service for Prometheus to collect them. To collectthese metrics,configure aPodMonitoring custom resource (CR).

The following table describes additional Hubble metrics:

To collect these additional metrics, you need to create aPodMonitoring resource in your cluster.

For example, to create aPodMonitoring resource that collects all the available Hubble metrics,complete the following steps:

1. Save the following manifest aspod-monitoring.yaml:

   apiVersion:monitoring.googleapis.com/v1kind:PodMonitoringmetadata:name:hubble-metricsnamespace:kube-system# The resource is created in the `kube-system` namespace because the Cilium pods that emit the metrics reside in that namespacespec:selector:matchLabels:k8s-app:ciliumendpoints:-port:9965interval:30spath:/metricsmetricRelabeling:-action:keepsourceLabels:[__name__]regex:(hubble_flows_processed_total|hubble_drop_total|hubble_port_distribution_total|hubble_tcp_flags_total|hubble_icmp_total)

2. Apply the manifest to your cluster by running the following command:

   kubectlapply-fpod-monitoring.yaml

   After applying this configuration, the additional Hubble metrics are available in Google Cloud Managed Service for Prometheus.

GKE Dataplane V2 observability tools

GKE Dataplane V2 observability provides a Managed Hubble solution with networkobservability and security insights for Kubernetes workloads deployed withGKE Dataplane V2.

When enabled, GKE Dataplane V2 observability deploys the following components toyour cluster:

• Hubble Relay: a service that collects network telemetry data about yourPods from each node.

• Hubble CLI: a command-line interface tool providing live trafficinformation within the cluster.

You can deploy the following component after you enable GKE Dataplane V2observability to your cluster:

• Hubble UI: a web-based tool that you can use to view and analyze thenetwork telemetry data that is collected by Hubble Relay. You must enableGKE Dataplane V2 observability to deploy Hubble UI.

How GKE Dataplane V2 metrics and observability works

GKE Dataplane V2 observability uses the following components and tools tocollect metrics and provide insights into your network traffic:

• GKE Dataplane V2: GKE Dataplane V2 metrics and observability useGKE Dataplane V2 datapath based oneBPF to collect metrics about traffic flows and network policy enforcement for a Podbased on a given workload.

• Google Cloud Managed Service for Prometheus:GKE Dataplane V2 metrics configures the Google Cloud Managed Service for Prometheus agent toingest aggregated metrics to Google Cloud Managed Service for Prometheus, a scalablemonitoring solution that can ingest and store large amounts of data that alsolets you build on theGoogle Cloud Observability.

• Hubble: GKE Dataplane V2 observabilityuses Hubble, an open source observability project. Hubble enables networkobservability and security insights for Kubernetes workloads deployed with aneBPF Dataplane.

  Hubble flow events occur when:

  • A network connection is first established

  • A TCP flag is first seen, which indicates the state of the TCP connection

  • A packet is transmitted after at least five seconds have passed since the lastflow event

  Hubble metrics: counts the number of flow events in a Kubernetes clusterthat you can use to identify which Pods are communicating with each other.

  Note: Hubble flow metrics don't count the amount of data or number ofpackets transmitted.

• Enabling metrics and observability: You can enable GKE Dataplane V2 andobservability independent of each other. To enable the Network topologyvisualization in the open source Hubble UI, you mustenable GKE Dataplane V2 observability.

• Autopilot clusters:

  • Metrics are enabled by default

  • Observability tools are disabled by default

  • You must create the ClusterPodMonitoring resource to gather metrics in theGoogle Cloud Managed Service for Prometheus

• Standard clusters:

  • Metrics are disabled by default

  • Observability tools are disabled by default

  • If you have Google Cloud Managed Service for Prometheus enabled, aPodMonitoring resourceis created automatically

  • APodMonitoring resource is marked asensure exists. You can stopsending metrics to Google Cloud Managed Service for Prometheus by editing thePodMonitoring resource to disable all metrics

GKE Dataplane V2 observability endpoints

GKE Dataplane V2 observability components expose the following two observabilityendpoints:

• Metrics endpoint: an HTTP endpoint that exposes traffic metrics inPrometheus format. Theanetd Pod exposes the metrics endpoint on each clusternode on port 9965.

• Flows port: a gRPC endpoint. Thehubble-relay Pod exposes the flows portendpoint as a KubernetesClusterIP Service on port 443. Thehubble-relayPod is the backend for the KubernetesClusterIP Service and all requests tothe KubernetesClusterIP Service are forwarded to thehubble-relay Pod.You can access the flow port using Hubble CLI or the Hubble UI.

Limitations

• GKE Dataplane V2 observability has a cluster-wide limit of 5000 nodes.
• GKE Dataplane V2 metrics and observability only work in clusters onGoogle Cloud withGKE Dataplane V2enabled.
• GKE Dataplane V2 metrics are similar to Hubble metrics in that they areimplemented as flow-based metrics to provide connection information. Thesemetrics don't count the amount of data nor number of packets transmitted.Given that the metrics are flow-based, they don't provide an accuraterepresentation of the amount of data transmitted in a network flow.

What's next

• Set up GKE Dataplane V2 observability
• Observe your traffic using GKE Dataplane V2 observability