Observability for proxyless gRPC

Cloud Service Mesh Observability for proxyless gRPC builds on top of theexisting gRPC OpenTelemetry plugin,records metrics (latency, message sizes, etc.) for all gRPC channels and serversthat are Cloud Service Mesh-enabled, and provides additional attributes thatshow topological mesh information for Cloud Service Mesh traffic. A gRPCchannel is considered to be Cloud Service Mesh enabled if it getsconfiguration from the Cloud Service Mesh control plane, while all gRPCservers are considered Cloud Service Mesh enabled.

Mesh Attributes

The following mesh attributes are available on metrics.

Local Environment Labels:

• csm.mesh_id
  • The mesh ID.
• Other local environment attributes are obtained from theOpenTelemetry Resource.
  • Managed Service for Prometheus (GMP)can be set up to use Google infrastructure to store metrics. If using this,resource attributes that describe the application's local environment areautomatically added in as aMonitoredResource.
  • If using non-Google infrastructure for exporting and storing metrics, thecollection pipeline should add in attributes on the metrics that describethe environment that the application is running on.

Remote Environment Labels:

• csm.remote_workload_type
  • The type of the remote peer. ("gcp_kubernetes_engine" for GKE).
• Based on the type of the peer, additional attributes would be present.
  • For a peer running on GKE -
  • csm.remote_workload_project_id
    • The identifier of the project associated with this resource, such as "my-project".
  • csm.remote_workload_location*The physical location of the cluster that contains the container.
  • csm.remote_workload_cluster_name
    • The cluster where the container is running.
  • csm.remote_workload_namespace_name
    • The namespace where the container is running.
  • csm.remote_workload_name
    • The name of the remote workload. This should be the name of the objectthat contains the Pod definition (for example, a Deployment, ReplicaSet,or just the Pod name for a bare Pod).

Service Labels: Information about the backend service (xDS cluster) that the RPCis being routed to. Note that this is only available if the backend service hasbeen configured through the Gateway API.

• csm.service_name
  • The service name.
• csm.service_namespace_name
  • The service namespace name.

The term remote_workload refers to the peer, that is, for clients, the serverPod that is the target of an RPC is the remote workload, whereas for servers,the client Pod that initiated the RPC is the remote workload.

Note that these attributes won't be available ongrpc.client.attempt.startedandgrpc.server.call.started since all topological mesh information is notavailable at the collection point of these metrics.

#include <grpcpp/ext/csm_observability.h>intmain(){// …autoobservability=grpc::CsmObservabilityBuilder().SetMeterProvider(std::move(meter_provider)).BuildAndRegister();assert(observability.ok());// …}

opentelemetry::exporter::metrics::PrometheusExporterOptionsopts;opts.url="0.0.0.0:9464";autoprometheus_exporter=opentelemetry::exporter::metrics::PrometheusExporterFactory::Create(opts);autometer_provider=std::make_shared<opentelemetry::sdk::metrics::MeterProvider>();meter_provider->AddMetricReader(std::move(prometheus_exporter));autoobservability=grpc:::CsmObservabilityBuilder().SetMeterProvider(std::move(meter_provider)).BuildAndRegister();

importio.grpc.gcp.csm.observability.CsmObservability;...publicstaticvoidmain(String[]args){...intprometheusPort=9464;SdkMeterProvidersdkMeterProvider=SdkMeterProvider.builder().registerMetricReader(PrometheusHttpServer.builder().setPort(prometheusPort).build()).build();OpenTelemetrySdkopenTelemetrySdk=OpenTelemetrySdk.builder().setMeterProvider(sdkMeterProvider).build();CsmObservabilityobservability=newCsmObservability.Builder().sdk(openTelemetrySdk).build();observability.registerGlobal();// ... (continue with channel and server configuration)}

import("context""google.golang.org/grpc/stats/opentelemetry""google.golang.org/grpc/stats/opentelemetry/csm""go.opentelemetry.io/otel/sdk/metric")funcmain(){reader:=metric.NewManualReader()provider:=metric.NewMeterProvider(metric.WithReader(reader))opts:=opentelemetry.Options{MetricsOptions:opentelemetry.MetricsOptions{MeterProvider:provider,},}cleanup:=csm.EnableObservability(context.Background(),opts)defercleanup()// Any created ClientConns and servers will be configured with an// OpenTelemetry stats plugin configured with provided options.}

grpcio>=1.65.0grpcio-observability>=1.65.0grpcio-csm-observability>=1.65.0

importgrpc_csm_observability# ...csm_plugin=grpc_csm_observability.CsmOpenTelemetryPlugin(meter_provider=[your_meter_provider],)csm_plugin.register_global()# Create server or client

csm_plugin.deregister_global()

importgrpc_csm_observabilityfromopentelemetry.exporter.prometheusimportPrometheusMetricReaderfromprometheus_clientimportstart_http_serverstart_http_server(port=9464,addr="0.0.0.0")reader=PrometheusMetricReader()meter_provider=MeterProvider(metric_readers=[reader])csm_plugin=CsmOpenTelemetryPlugin(meter_provider=meter_provider,)csm_plugin.register_global()# Clean up after usecsm_plugin.deregister_global()

In the previous sample, you can scrapelocalhost:9464/metrics to get themetrics reported by Cloud Service Mesh Observability.

Note that for the mesh attributes added onto the gRPC metrics to work, both theclient and server binaries need to be set up with CsmObservability.

If using non-Google infrastructure for exporting and storing metrics, thecollection pipeline should add in attributes on the metrics that describe theenvironment that the application is running on. This along with the meshattributes described previously can be utilized to get a view of the trafficrunning on the mesh.

Spec changes

Cloud Service Mesh Observability determines the mesh topological informationthrough environment variables that need to be added to the container's env, bothfor clients and servers. This information is made available to peers for metricsreporting through Cloud Service Mesh Observability.

spec:containers:-image:IMAGE_NAMEname:CONTAINER_NAMEenv:-name:GRPC_XDS_BOOTSTRAPvalue:"/tmp/grpc-xds/td-grpc-bootstrap.json"#created by td-grpc-bootstrap-name:POD_NAMEvalueFrom:fieldRef:fieldPath:metadata.name-name:NAMESPACE_NAMEvalueFrom:fieldRef:fieldPath:metadata.namespace-name:CSM_WORKLOAD_NAMEvalue:CSM_WORKLOAD_NAME-name:CONTAINER_NAMEvalue:CONTAINER_NAME-name:OTEL_RESOURCE_ATTRIBUTESvalue:k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME),k8s.container.name=CONTAINER_NAME

Replacing the following:

• IMAGE_NAME with the name of the image.
• CONTAINER_NAME with the name of the container.
• CSM_WORKLOAD_NAME with the workload name, for example the deploymentname.