This document describes how to modify a Python app to collect traceand metric data using the open sourceOpenTelemetry framework, and howto write structured JSON logs to standard out. This document also providesinformation about a sample Python app that you can install and run. The appuses theFlask web framework and is configured to generate metrics,traces, and logs.

To learn more about instrumentation, see the following documents:

• Instrumentation and observability.
• Choose an instrumentation approach.

This document doesn't describe zero-code instrumentation. For information aboutthat topic, seePython zero-code instrumentation.

For general information, seeOpenTelemetry Instrumentation for Python.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com

1. If you run the sample in the Cloud Shell, on Google Cloudresources, or on a local development environment, then the permissions listedin this section are sufficient. For production applications, typically aservice account provides the credentials to write log, metric, and tracedata.

   To get the permissions that you need to have the sample application to write log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Writer (roles/logging.logWriter)
   • Monitoring Metric Writer (roles/monitoring.metricWriter)
   • Cloud Trace Agent (roles/cloudtrace.agent)

   To get the permissions that you need to view your log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Viewer (roles/logging.viewer)
   • Monitoring Viewer (roles/monitoring.viewer)
   • Cloud Trace User (roles/cloudtrace.user)

   For more information about granting roles, seeManage access to projects, folders, and organizations.

   You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Instrument your app to collect traces, metrics, and logs

To instrument your app to collect trace and metric data and to writestructured JSON to standard out, perform the following steps as describedin subsequent sections of this document:

1. Configure OpenTelemetry
2. Configure structured logging

Configure OpenTelemetry

This example app is configured to use the OpenTelemetry Python SDK to export tracesand metrics by using theOTLP protocol. By default, the OpenTelemetryPython SDK uses theW3C Trace Context format forpropagating trace context, which ensures that spans have thecorrect parent-child relationship within a trace.

The following code sample illustrates a Python module to setup OpenTelemetry.To view the full sample, clickmore_vertMore,and then selectView on GitHub.

defsetup_opentelemetry()->None:resource=Resource.create(attributes={# Use the PID as the service.instance.id to avoid duplicate timeseries# from different Gunicorn worker processes.SERVICE_INSTANCE_ID:f"worker-{os.getpid()}",})# Set up OpenTelemetry Python SDKtracer_provider=TracerProvider(resource=resource)tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))trace.set_tracer_provider(tracer_provider)logger_provider=LoggerProvider(resource=resource)logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))logs.set_logger_provider(logger_provider)reader=PeriodicExportingMetricReader(OTLPMetricExporter())meter_provider=MeterProvider(metric_readers=[reader],resource=resource)metrics.set_meter_provider(meter_provider)

The Flask app relies onGunicorn to serve HTTP requests following therecommendations in Flask'sDeploying to Production guide.Gunicorn starts multiple copies of your app running in independent worker processesto increase throughput. To ensure that metrics from the worker processes don't conflictwith each other, we recommend that each worker process sets a unique value for theservice.instance.id resource attribute. One way to do this is by including theprocess ID in theservice.instance.id. For more information, seeTime-series collisions.

For more information and configuration options, seeOpenTelemetry Pythoninstrumentation.

Configure structured logging

To write structured logs that are linked to traces, configure your app tooutput JSON-formatted logs to standard out with keys containing traceinformation. The following code sample illustrates how to configure the standardlogging library to output JSON structured logs using thepython-json-logger library, and how to use theopentelemetry-instrumentation-logging package to include traceinformation.

classJsonFormatter(jsonlogger.JsonFormatter):defformatTime(self,record:logging.LogRecord,datefmt:Optional[str]=None):# Format the timestamp as RFC 3339 with microsecond precisionisoformat=datetime.fromtimestamp(record.created).isoformat()returnf"{isoformat}Z"defsetup_structured_logging()->None:LoggingInstrumentor().instrument()log_handler=logging.StreamHandler()formatter=JsonFormatter("%(asctime)s%(levelname)s%(message)s%(otelTraceID)s%(otelSpanID)s%(otelTraceSampled)s",rename_fields={"levelname":"severity","asctime":"timestamp","otelTraceID":"logging.googleapis.com/trace","otelSpanID":"logging.googleapis.com/spanId","otelTraceSampled":"logging.googleapis.com/trace_sampled",},)log_handler.setFormatter(formatter)logging.basicConfig(level=logging.INFO,handlers=[log_handler],)

The previous configuration extracts information about the active span from thelog message, and then adds that information as attributes to the JSON structuredlog. These attributes can then be used to correlate a log with a trace:

• logging.googleapis.com/trace: Resource name of the trace associated withthe log entry.
• logging.googleapis.com/spanId: The span ID with the trace that isassociated with the log entry.
• logging.googleapis.com/trace_sampled: The value of this field must betrue orfalse.

For more information about these fields, see theLogEntrystructure.

Run a sample app configured to collect telemetry

The example app uses vendor-neutral formats, including JSON for logs and OTLPfor metrics and traces. Telemetry from the app is routed to Google Cloud usingthe OpenTelemetryCollector configured with Google exporters. It usesFlask to serve HTTP requests, and therequests library formaking HTTP requests. To generate metrics and traces for the HTTP client andserver, the example app installs theopentelemetry-instrumentation-flask andopentelemetry-instrumentation-requestsinstrumentation libraries:

logger=logging.getLogger(__name__)# Initialize OpenTelemetry Python SDK and structured loggingsetup_opentelemetry()setup_structured_logging()app=Flask(__name__)# Add instrumentationFlaskInstrumentor().instrument_app(app)RequestsInstrumentor().instrument()

The app has two endpoints:

• The/multi endpoint is handled by themulti function. The loadgenerator in the app issues requests to the/multi endpoint. When thisendpoint receives a request, it sends between three and seven requests tothe/single endpoint on the local server.

  @app.route("/multi")defmulti():"""Handle an http request by making 3-7 http requests to the /single endpoint."""sub_requests=randint(3,7)logger.info("handle /multi request",extra={"subRequests":sub_requests})for_inrange(sub_requests):requests.get(url_for("single",_external=True))return"ok"

• The/single endpoint is handled by thesingle function. When thisendpoint receives a request, it sleeps for a short delay and then respondswith a string.

  @app.route("/single")defsingle():"""Handle an http request by sleeping for 100-200 ms, and write the number of seconds slept as the response."""duration=uniform(0.1,0.2)logger.info("handle /single request",extra={"duration":duration})time.sleep(duration)returnf"slept{duration} seconds"

Download and deploy the app

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Clone the repository:

   gitclonehttps://github.com/GoogleCloudPlatform/opentelemetry-operations-python

3. Go to the sample directory:

   cdopentelemetry-operations-python/samples/instrumentation-quickstart

4. Build and run the sample:

   dockercomposeup--abort-on-container-exit

   If you aren't running on Cloud Shell, then run the application with theGOOGLE_APPLICATION_CREDENTIALS environment variable pointing to acredentials file.Application DefaultCredentialsprovides a credentials file at$HOME/.config/gcloud/application_default_credentials.json.

   # Set environment variablesexportGOOGLE_CLOUD_PROJECT="PROJECT_ID"exportGOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"exportUSERID="$(id-u)"# Rundockercompose-fdocker-compose.yaml-fdocker-compose.creds.yamlup--abort-on-container-exit

View your metrics

The OpenTelemetry instrumentation in the sample app generates Prometheusmetrics that you can view by using theMetrics Explorer:

• Prometheus/http_server_duration_milliseconds/histogramrecords the duration of server requests and stores the resultsin a histogram.

• Prometheus/http_client_duration_milliseconds/histogramrecords the duration of client requests and stores the resultsin a histogram.

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterhttp_server in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectPrometheus Target.
   2. In theActive metric categories menu, selectHttp.
   3. In theActive metrics menu, select a metric.
   4. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. Configure how the data is viewed.
   

   When the measurements for a metric arecumulative, Metrics Explorer automatically normalizes the measured data bythe alignment period, which which results in the chart displaying a rate. Formore information, seeKinds, types, and conversions.

   When integer or double values are measured, such as with the twocounter metrics, Metrics Explorer automatically sums all time series.To view the data for the/multi and/single HTTP routes,set the first menu of theAggregation entry toNone.

   For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

View your traces

It might take several minutes before your trace data is available. For example,when trace data is received by your project, Google Cloud Observability might need to createa database to store that data. The creation of the database can take a fewminutes and during that period, no trace data is available to view.

To view your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

2. In the table section of the page, select a row with the span name/multi.

3. In the Gantt chart on theTrace details panel,select the span labeled/multi.

   A panel opens that displays information about the HTTP request. Thesedetails include the method, status code, number of bytes, and theuser agent of the caller.

4. To view the logs associated with this trace,select theLogs & Events tab.

   The tab shows individual logs. To view the details of the log entry,expand the log entry. You can also clickView Logs and view the logby using the Logs Explorer.

For more information about using the Cloud Trace explorer, seeFind and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can alsoview associated traces, when they exist.

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. Locate a log with the description ofhandle /multi request.

   To view the details of the log, expand the log entry.

3. ClickTraces on a log entry with the"handle /multi request" message, and then selectView trace details.

   ATrace details panel opens and displays the selected trace.

   Your log data might be available several minutes before your tracedata is available. If you encounter an error when viewing trace dataeither by searching for a trace by ID or by following the steps inthis task, then wait a minute or two and retry the action.

For more information about using the Logs Explorer, seeView logs by using the Logs Explorer.

What's next

• OpenTelemetry
• OTLP specification
• Structured logging
• Troubleshooting Managed Service for Prometheus
• Troubleshoot Cloud Trace