Observability with Envoy

This document demonstrates how to generate tracing and logging for the Envoyproxy. It also shows you how to export the information toCloud Trace andCloud Logging.

Using a service mesh gives you the ability to observe traffic to and fromservices, which allows for richer monitoring and debugging without code changesin the service itself. In the sidecar proxy architecture thatCloud Service Mesh uses, the proxy is the component that processes requestsand provides the necessary telemetry information. Telemetry information must becollected and stored in a centralized location for further use, such as dataanalysis, alerting, and troubleshooting.

The following steps correspond to the numbered labels in the diagram:

1. Cloud Service Mesh configures the Envoy proxy to do the following:

   • Load balance traffic for the10.10.10.10:80 service.
   • Store access log information for each request issued for this service.
   • Generate tracing information for the service.

2. After the consumer sends a request to10.10.10.10, the sidecar proxyroutes the request to the correct destination.

3. The sidecar proxy also generates the necessary telemetry information:

   1. Adds an entry to the access log on the local disk withadditional information about the request.
   2. Generates a trace entry and sends it to Traceby using OpenCensus Envoy tracing.

4. The Logging agent exports this data to the Cloud Logging APIso that the data becomes available in the Cloud Logging interface.

Prerequisites

Before you complete the setup steps, ensure that the following is done:

1. The Traffic Director API is enabled and other prerequisites are met, as described inPrepare to set up with VM and proxyless workloads.
2. TheCloud Trace API is enabled.
3. The service account that the Compute Engine VM uses has thefollowing Identity and Access Management (IAM) roles configured:
   • Cloud Trace Agent role(roles/cloudtrace.agent)
   • Logs Writer role(roles/logging.logWriter)
4. Thefirewall rules allow traffic to the VM that youconfigure as part of this setup.

Set up the demonstration service and Cloud Service Mesh

This document uses several shell scripts to perform the steps required toconfigure the demonstration service. Review the scripts to understand thespecific steps that they perform.

1. Start a Compute Engine VM and configure the HTTP service on the VM:

   curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_service.shchmod 755 setup_demo_service.sh && ./setup_demo_service.sh

   Thesetup_demo_service.sh script creates a VM template that launchesapache2 when a VM starts and a managed instance group that uses thistemplate. The script launches a single instance without autoscaling enabled.

2. Use Cloud Service Mesh to configure routing for the10.10.10.10 service:

   curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_trafficdirector.shchmod 755 setup_demo_trafficdirector.sh && ./setup_demo_trafficdirector.sh

   Thesetup_demo_trafficdirector.sh script configures the necessaryparameters for the Cloud Service Mesh managed service.

3. Start a Compute Engine VM that runs a consumer of the HTTPservice, with the sidecar proxy installed and configured on the VM. In thefollowing command, replacePROJECT_ID with the projectID to which Trace information should be sent. This istypically the same Google Cloud project to which your VM belongs.

   curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_client.shchmod 755 setup_demo_client.sh && ./setup_demo_client.shPROJECT_ID

   Thesetup_demo_client.sh script creates a Compute Engine VM that has anEnvoy proxy preconfigured to use Cloud Service Mesh.

The following additional configuration settings enable tracing and logging:

• TheTRAFFICDIRECTOR_ACCESS_LOG_PATH andTRAFFICDIRECTOR_ENABLE_TRACINGbootstrap node metadata variables enable logging and tracing, as described inConfigure Envoy bootstrap attributes for Cloud Service Mesh.
• Static bootstrap configuration enables export of trace information toTrace by using OpenCensus.

After running these scripts, you can sign in to thetd-observability-demo-clientVM and access the HTTP service available at10.10.10.10:

curl http://10.10.10.10

At this point, Envoy generates access logging and tracing information. Thefollowing section describes how to export logs and tracing information.

Set up trace export to Cloud Trace

The Envoy bootstrap configuration that you created when you ran thesetup-demo-client.sh script is sufficient to generate tracing information.All other configuration is optional. If you want to configure additionalparameters, see theOpenCensus Envoy configuration page and modify the tracing options in the Envoy bootstrap configuration.

After you issue a sample request to the demonstration server (curl10.10.10.10), in the Google Cloud console, go to the Trace interface(Trace> Trace list).You see a trace record that corresponds to the request that you issued.

For more information about how to use Trace, see theCloud Trace documentation.

Set up access log export to Logging

At this stage, Envoy should be recording access log information to the localdisk of the VM where it is running. Toexport these records toLogging, youmust install the Logging agent locally. Thisrequires installing and configuring the Logging agent.

Install the Logging agent

Install the Logging agent on the VM from whichlogging information is exported. For this example configuration, the VM istd-observability-demo-vm.

curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.shsudo bash add-logging-agent-repo.sh --also-install

For more information, seeInstall the Cloud Logging agent on a single VM.

Configure the Logging agent

You can export the Envoy logs as either unstructured or structured text.

Export the Envoy logs as unstructured text

This option exports log records from the access log to Cloud Loggingas raw text. Each entry in the access log is exported as a single entryto Logging. This configuration is easier toinstall because it relies on a parser that is distributed with the currentversion of the Logging agent. However, it is moredifficult to filter and process raw text log entries when using this option.

1. Download and install the Envoy access log unstructured export configurationfile:

   curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/envoy_access_fluentd_unstructured.confsudo cp envoy_access_fluentd_unstructured.conf /etc/google-fluentd/config.d/envoy_access.conf

2. Restart the agent; the changes take effect when the agent starts up:

   sudo service google-fluentd restart

Export the Envoy logs as structured text

1. Install the Envoy access log parser from GitHub:

   sudo /opt/google-fluentd/embedded/bin/gem install fluent-plugin-envoy-parser

2. Download and install the configuration file for exporting Envoy access logsin a structured format:

   curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/envoy_access_fluentd_structured.confsudo cp envoy_access_fluentd_structured.conf /etc/google-fluentd/config.d/envoy_access.conf

3. Restart the agent; the changes take effect when the agent starts up:

   sudo service google-fluentd restart

For more information, seeConfigure the Logging agent.

Verify the configuration

1. From the sidecar proxy VM, generate a request to the demonstration service.This creates a new local log record. For example, you can runcurl 10.10.10.10.
2. In the Google Cloud console, go toLogging> Logs Explorer.In the drop-down menu, select theenvoy-access log type. You see alog entry for the most recent request in the unstructured or structuredformat, depending on the configuration type that you chose earlier.

Troubleshooting

Read the following sections for information about how to troubleshoot differentissues with observability with Envoy.

Configure tracing across multiple projects

If you would like to trace requests across Envoys deployed in multipleprojects, note the following:

• Each Envoy must be configured with the credentials of the project where itis running.
• Each Envoy sends trace data to the project that corresponds to the credentialsit is running with.
• You can see tracing spans for cross-project requests if your applicationspreserve the value of theX-Cloud-Trace-Context HTTP headerwhen requests are made.

Trace compatibility with proxyless gRPC applications

Envoy's OpenCensus tracer configuration allows traces exported from proxylessgRPC applications and Envoy proxies to be fully compatible within a servicemesh. For compatibility, the Envoy bootstrap must configure the trace context toinclude theGRPC_TRACE_BIN trace format in itsOpenCensusConfig, as follows:

tracing:  http:      name: envoy.tracers.opencensus      typed_config:        "@type": type.googleapis.com/envoy.config.trace.v2.OpenCensusConfig        stackdriver_exporter_enabled: "true"        stackdriver_project_id: "PROJECT_ID"        incoming_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]        outgoing_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]

If configuration is complete, but you don't see trace or logging entriesavailable, verify the following:

1. The service accounts for the Compute Engine VM have the necessaryTrace and Logging IAMpermissions, as specified in the prerequisites. For information aboutTrace IAM permissions, seeAccess control.For information about Logging permissions, seeAccess control.
2. For logging: Ensure that there are no errors in/var/log/google-fluentd/google-fluentd.log.
3. For logging: Ensure that new entries appear in the local access log filewhen requests are issued.

What's next

• To find related information, seeObservability with proxyless gRPC applications.
• To find general Cloud Service Mesh troubleshooting information, seeTroubleshooting deployments that use Envoy.