This pageapplies toApigee andApigee hybrid.

View Apigee Edge documentation.

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This page demonstrates the steps required to configure distributed tracing for your Apigee runtime. If you are new to using distributed tracing systems and would like more information, seeUnderstanding Distributed Tracing.

Introduction

Distributed tracing systems let you track a request in a software system distributed acrossmultiple applications, services, and databases, as well as intermediaries like proxies. These tracingsystems generate reports showing the time taken by a request at each step. Tracing reports can alsoprovide a granular view of the various services called during a request, enabling a deeper understanding of what happens at each step in your software system.

Thetrace tool in Apigee Edge and thedebug tool in Apigee are useful for troubleshooting andmonitoring your API proxies. However, these tools do not send any data to distributedtracing servers likeCloud Trace orJaeger.To view Apigee runtime data in a distributed tracing report, you mustexplicitly enable distributed tracing in your Apigee runtime. Once tracing is enabled, theruntime can send trace data to distributed tracing servers and participate in an existing trace. As a result, you can view data from inside and outside of your Apigee ecosystem from a single location.

You can view the following information in distributed tracing reports:

• Execution time of an entire flow.
• Time at which the request is received.
• Time at which the request is sent to the target.
• Time at which the response is received from the target.
• Execution time of each policy in a flow.
• Execution time of service callouts and target flows.
• Time at which the response is sent to the client.

In the distributed tracing report, you can view the execution details of the flows asspans. A span refers to the time taken by a flow in a trace. The time taken to execute a flow is displayed as an aggregate of the time required to execute each policy in the flow. You can view each of the following flows as individual spans:

• Request
  • Proxy
    • Preflow
    • PostFlow
  • Target
    • Preflow
    • PostFlow
• Response
  • Proxy
    • Preflow
    • PostFlow
  • Target
    • Preflow
    • PostFlow

Once you enable distributed tracing, the Apigee runtime will trace a set of predefined variables by default. For more information, seeDefault trace variables in tracing report. You can use the TraceCapture policy to extend the default runtime behavior and trace additional flow, policy, or custom variables. For more information, see theTraceCapture policy.

Because tracing all the API calls in Apigee's runtime would impact performance, Apigee lets you configure a probabilistic sampling rate. By using the sampling rate, you can specify the number of API calls that are sent for distributed tracing. For example, if you specify the sampling rate as0.4, it means that 40% of the API calls are sent for tracing. For more information, seePerformance considerations.

Both the Apigee runtime and the Apigee hybrid runtime support distributed tracing using Cloud Trace. If you are using Jaeger, you can skip this section and proceed toEnabling distributed tracing for Jaeger.

Configure Apigee runtime for Cloud Trace

To use Cloud Trace with an Apigee runtime, your Google Cloud project must have theCloud Trace API enabled. This setting lets your Google Cloud project receive trace data from authenticated sources.

To confirm that the Cloud Trace API is enabled do the following:

1. From the Google Cloud console, go toAPIs and Services:

   Go to APIs and Services

2. ClickEnable APIs and Services.
3. In the search bar, enterTrace API.
4. IfAPI enabled is displayed, this API is already enabled and there is nothing for you to do. Otherwise, clickEnable.

Configure Apigee hybrid runtime for Cloud Trace

The Cloud Trace API must be enabled to use Cloud Trace with an Apigee hybrid runtime. To confirm that the Cloud Trace API is enabled, follow the steps inConfigure Apigee runtime for Cloud Trace.

In addition to enabling the Cloud Trace API, you must add theiam.gserviceaccount.com service account to use Cloud Trace with the hybrid runtime. To add the serviceaccount, along with the require roles and keys, perform the following steps:

1. Create a new service account:

   gcloud iam service-accounts create \    apigee-runtime --display-name "Service Account Apigee hybrid runtime" \    --projectPROJECT_ID

2. Add an IAM policy binding to the service account:

   gcloud projects add-iam-policy-binding \PROJECT_ID --member "serviceAccount:apigee-runtime@PROJECT_ID.iam.gserviceaccount.com" \    --role=roles/cloudtrace.agent --projectPROJECT_ID

3. Create a service account key:

   gcloud iam service-accounts keys \    create ~/apigee-runtime.json --iam-account apigee-runtime@PROJECT_ID.iam.gserviceaccount.com

   Service account keys are a security risk if not managed correctly. You are responsible for thesecurity of the private key and for other operations described by Best practices for managing service account keys.If you are prevented from creating a service account key, service account key creation mightbe disabled for your organization. For more information, see Managing secure-by-default organization resources.

   If you acquired the service account key from an external source, you must validate it before use. For more information, see Security requirements for externally sourced credentials.

4. Add the service account to theoverrides.yaml file.

envs: - name: ENV_NAME   serviceAccountPaths:   runtime: apigee-runtime.json   synchronizer: apigee-sync.json   udca: apigee-udca.json

5. Apply the changes to the runtime usingHelm:

   helm upgradeENV_NAME apigee-env/ \    --namespaceAPIGEE_NAMESPACE \    --set env=ENV_NAME \    --atomic \    -foverrides.yaml

Enable distributed tracing

Before enabling distributed tracing for Cloud Trace or Jaeger, create the following environment variables:

TOKEN="Authorization: Bearer $(gcloud auth print-access-token)"ENV_NAME=YOUR_ENVIRONMENT_NAMEPROJECT_ID=YOUR_GOOGLE_CLOUD_PROJECT_ID

Where:

• TOKEN defines the Authentication header with a bearer token. You use this header when calling Apigee APIs. For more information, see the reference page for theprint-access-token command.
• ENV_NAME is the name of an environment in your organization.
• PROJECT_ID is the ID of your Google Cloud project.

Enable distributed tracing for Cloud Trace

The following example shows you how to enable the distributed tracing for Cloud Trace:

1. Execute this Apigee API call:

   curl -H "$TOKEN" \    -H "Content-Type: application/json" \    https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/$ENV_NAME/traceConfig \    -X PATCH \    -d '{"exporter":"CLOUD_TRACE","endpoint": "'"$PROJECT_ID"'",    "samplingConfig": {"sampler": "PROBABILITY","samplingRate": 0.1}}'

   The example request body consists of the following elements:

   • ThesamplingRate is set to 0.1. This means approximately 10% of the API calls are sent to distributed tracing. For more information on setting asamplingRate for your runtime environment, seePerformance considerations.
   • Theexporter parameter is set toCLOUD_TRACE.
   • The endpoint is set to the Google Cloud project where you want the trace to be sent. NOTE: This must match the service account that was done in the configuration step.

   A successful response looks similar to the following:

   {  "exporter": "CLOUD_TRACE",  "endpoint": "staging",  "samplingConfig": {    "sampler": "PROBABILITY",    "samplingRate": 0.1  }}

Enable distributed tracing for Jaeger

The following example shows you how to enable the distributed tracing for Jaeger:

curl -s -H "$TOKEN" \    'https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/$ENV_NAME/traceConfig' \    -X PATCH \    -H "content-type:application/json" -d '{    "samplingConfig": {    "samplingRate": 0.4,    "sampler": "PROBABILITY"},    "endpoint": "http://DOMAIN:9411/api/v2/spans",    "exporter": "JAEGER"    }'

In this example:

• ThesamplingRate is set to 0.4. This means approximately 40% of the API calls are sent to distributed tracing.
• Theexporter parameter is set toJAEGER.
• The endpoint is set to where Jaeger is installed and configured.

When you run the command, you can see a response similar to the following:

{  "exporter": "JAEGER",  "endpoint": "staging",  "samplingConfig": {    "sampler": "PROBABILITY",    "samplingRate": 0.4  }}

View the distributed tracing configuration

To view the existing distributed tracing configuration in your runtime, log in to your runtime and then run the following command:

curl -H "$TOKEN" \    -H "Content-Type: application/json" \    https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/$ENV_NAME/traceConfig

When you run the command, you can see a response similar to the following:

{  "exporter": "CLOUD_TRACE",  "endpoint": "staging",  "samplingConfig": {    "sampler": "PROBABILITY",    "samplingRate": 0.1  }}

Update the distributed tracing configuration

The following command shows you how to update the existing distributed tracing configuration for Cloud Trace:

curl -s \    -H "$TOKEN" \    'https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/$ENV_NAME/traceConfig?updateMask=endpoint,samplingConfig,exporter' \    -X PATCH -H "content-type:application/json" \    -d '{"samplingConfig": {"samplingRate": 0.05, "sampler":"PROBABILITY"},    "endpoint":"staging", exporter:"CLOUD_TRACE"}'

When you run the command, you can see a response similar to the following:

{  "exporter": "CLOUD_TRACE",  "endpoint": "staging",  "samplingConfig": {    "sampler": "PROBABILITY",    "samplingRate": 0.05  }}

curl -H "$TOKEN" \    -H "Content-Type: application/json" \    https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/$ENV_NAME/traceConfig \    -X PATCH -d '{"exporter": "CLOUD_TRACE","endpoint": "'"$PROJECT_ID"'","samplingConfig":    {"sampler": "OFF","samplingRate": 0.1}}'

{  "exporter": "CLOUD_TRACE",  "endpoint": "staging",  "samplingConfig": {    "sampler": "OFF",    "samplingRate": 0.1  }}

curl -s -H "$TOKEN" \     'https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/ENV_NAME/traceConfig/overrides' \     -X POST \     -H "content-type:application/json" \     -d '{"apiProxy": "hello-world","samplingConfig": {"sampler": "PROBABILITY","samplingRate": 0.1}}'