To view an aggregate representation of your trace data, or to find and exploreindividual traces or traces that contain specific labels, use theTrace Explorer page.

The following screenshot illustrates how trace data is shown:

To run queries that perform an aggregate analysis of your spans, use theLog Analytics page or BigQuery. Both interfaces support queryingyour span data by using SQL. When you use theLog Analytics page, you canview query results as a table or as a chart. To learn more about usingSQL to query your trace data, seeQuery and analyze traces.

This feature is supported only for Google Cloud projects.ForApp Hubconfigurations, select the App Hub host project or management project.

About theTrace Explorer page

To help you identify trends and patterns in your trace data, the latency datais aggregated and displayed in charts. The heatmap uses color to representthe number of spans in a cell. A cell with many spans has a darker color thanone with few spans. Each cell's tooltip displays latency and other information.You can also view latency percentiles and span rate information.

When you are investigating an issue, you might want to view a specific traceor only spans with certain properties:

• To restrict which spans are shown,apply filters. Forexample, you can filter by service, application, error state, and otherproperties.

• To display a specific trace, use the toolbar'sclickpageviewSearch for trace option.

To identify outliers and view details of individual spans, use the tabularsection:

• You can sort the rows of theSpans andGrouped tabs by the durationheader.

• To understand how many spans different services are generating, use theGrouped tab. Each row corresponds to a service and span name, andfor each entry, latency statistics are shown.

To explore details of a span or grouping, select the entry from a row in thetabular section. A flyout opens that displays the span in the context of itstrace. From this view, you can explore the span, its attributes, and view eventsand log data. You also cansearch the spans and attributes in the trace for keywords.

The trace data that theTrace Explorer page depends on thefollowing:

• The projects searched for trace data. By default, the system only searchesthe project selected by the project picker for trace data. However, you canconfigure the page to search the list of projects in atrace scope.
• Your Identity and Access Management (IAM) permissions on the searched projects.If you don't have the permission to view trace data for a project, thenthe Google Cloud console displays a warning message and the data from thatproject isn't displayed.
• The time-range setting.
• The filters you apply.

The remainder of this page provides more information about how to find andexplore your trace data.

Before you begin

To get the permissions that you need to view trace data by using the Google Cloud console and to or select a trace scope, ask your administrator to grant you theCloud Trace User (roles/cloudtrace.user) IAM role on your project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to view trace data by using the Google Cloud console and to or select a trace scope. To see the exact permissions that are required, expand theRequired permissions section:

Required permissions

The following permissions are required to view trace data by using the Google Cloud console and to or select a trace scope:

• To select a trace scope:cloudtrace.traceScopes.[get, list]
• To read the default trace scope: observability.scopes.get

You might also be able to get these permissions withcustom roles or otherpredefined roles.

For more information about roles, seeControl access with Identity and Access Management.

View aggregated trace data

To view the aggregated information about your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

   It might take several minutes after thefirst trace data is writtento a Google Cloud project before that data is available to view.If you don't see any trace data displayed after waiting a few minutes, thenyour project might not have any data to display or there might be aconfiguration issue. For information about how to resolve these issues, seeTroubleshoot: No data in the Trace interface.

2. Configure theTrace Explorer page:

   1. In the toolbar of the Google Cloud console,select your Google Cloud project. ForApp Hubconfigurations, select the App Hub host project or management project.

   2. Optional: Configurewhich projects are searched for trace data by using theScope element:

      • To show the trace data that is stored in your project, set thefirst menu of theScope element toProject or to_Default.These two settings are equivalent.

      • To show the trace data that is stored in multiple projects, expand thefirst menu of theScope element, selectTrace scope, and then select the trace scope that liststhose projects. After you make your selection, theScope menu displaysa trace scope icon,, and the name of theselected trace scope.

      The data that is returned depends on yourIAM roles on the searched projects. For example, thesearched projects include a Google Cloud project that you don't haveaccess to, then no trace data for that project is returned.

      For more information, seeCreate and manage trace scopes.

   3. Optional: Update the time range by using the time-range selector or byusing your pointer to highlight a range on the x-axis.

      For example, you might set this selector toLast 2 weeks when you wantto see whether there are any trends in the latency data.Span data is stored for 30 days.

3. Explore the heatmap:

   • Use color to identify trends and outliers. A cell with many spans has a darker color than one with few spans.

   • View details of the cell by using the cell's tooltip. The tooltip lists number of spans, date and time, and the time interval of the cell.

     Note: If you select a cell in the heatmap, then the table that listsindividual spans or groupings of spans is updated.

4. Explore latency percentiles and error rates:

   • To view latency trends, set theChart view menu toSpan duration (percentile).The duration chart displays the 50th, 90th, 95th, and 99th percentiles.

   • To view the response status as a function of time,set theChart view menu toSpan rate. The chart displays therate of spans being sent to your project.

5. Use theSpans andGrouped tables to identify outliers or to viewstatistical information. TheSpans table lists individual spans. TheGrouped table groups spans by their name and by service or workload name.

   • To view outliers or find a span name, sort the table. You can sort byany column header.

     For example, sorting theSpans table by duration lets you findspans with high latency.

   • To view statistics like average latency, use theGrouped table.

   • To find App Hub-registered services and workloads, searchtheService/workload column for entries that include theservice icon,, or theworkload icon,.The link provided with an App Hub-registeredservice or workload opens the corresponding Application Monitoring dashboard.

   If you want more information about a particular grouping, then select thespan name for a row. A flyout opens that displays the span in the context ofits trace. You can thenexplore the span and trace.

6. To restrict which spans are shown, add filters.To learn more about filtering your trace data, see the next section.

Filter your trace data

To display only information that is of interest to you, apply filters.Filters restrict what data is shown. For example, you mightfilter by service name and by status. Or, if you've deployed applicationstoApp Hub, you might want to view the trace dataonly for the application, or for a specific service or workload that is partof the application.

When you add or remove a filter, the data displayed by theTrace Explorer page is refreshed andonly shows those spans that match all applied filters.

Filter by a trace by ID

When you are troubleshooting an incident or failure, you might know thetrace ID. To explore that trace, go to the toolbar,clickpageviewSearch for trace,and enter thetrace ID.

When you enter a valid ID, theDetails flyout opens and displaysinformation about the trace and its spans. You can use the options in thatpane to explore the trace. For example, you mightsearch the spans for keywords.

Note: If you enter a trace ID that isn't found, then the trace data might bestored in a project that isn't searched by theTrace Explorer page,or you might not have an IAM role that lets youview trace data. To resolve these failures, identify the projects thatstore the trace data. For each project, verify that yourIAM role lets you view trace data and then ensure that theScope element is configured to search allprojects that store your trace data. For more information, seeUpdate the list of projects searched for trace data.

Apply span filters

TheSpan filters pane lists the most common filters. You can selectmultiple entries from any subcategory. As you add or remove filters,theFilter bar is also updated.

The values for all menus are derived from your trace data. When a menu includesan option without any text, that option refers to spans that don'tinclude the corresponding attribute.

The followingSpan filters are typically available:

• OpenTelemetry service: Filters by theservice.name attribute.
• Span name: Name of the span.
• Span status: The request status. For information about the values,see theOpenTelemetry SpanStatus documentation.
• Duration: The duration of the span.
• Span kind: Describes the relationships between spans. For informationabout the values, see theOpenTelemetry SpanKind documentation.
• App Hub application: Filters by thegcp.apphub.application.idresource attribute.
• App Hub service: Filters by thegcp.apphub.service.idresource attribute.
• App Hub workload: Filters by thegcp.apphub.workload.idresource attribute.

If you want to filter by an attribute that isn't listed in theSpan filters pane, then use theFilter bar.

Use the filter bar

TheFilter bar lets you apply a filter with a predefined filter key anda value that you select, or you can enter both the key and value.

To add a filter, selectAdd filter, and then do one of the following:

• Select a defined key, likeSpan name, and then select a value fromthe secondary menu.

• SelectAdd attribute filter, and then add your custom key and value.If you enter your own filter key, then use the same syntax as a key for anattribute on a span.

  For example, to filter by the host identifier, set the key tohost.id.Similarly, to filter by status code, set the key to/http/status_code.In this scenario, you might set the value to200, which results in thefilter/http/status_code: 200.To let the filter match any value, selectAny value.

Filter by application

To find or view spans yourApp Hub-registered services andworkloads generate, your spans must contain the following resource attributes:

• gcp.apphub.application.{container,id,location}
• gcp.apphub.{workload,service}.{criticality_type,environment_type,id}

The previous resource attributes are generated byinstrumentation that you add to your applications.You can filter your trace data by any resource attribute.However, the followingSpan filters filter by the application, service,and workload IDs:

• App Hub application
• App Hub service
• App Hub workload

To open the Application Monitoring dashboard for aApp Hub-registered service and workload, do the following:

1. Go to theSpans orGrouped table and find the service orworkload. You might look for the service icon,,the workload icon,, or filter the table.

2. Select the link in theService/workload column.

Explore a trace

This section provides information about how you can explore a span in thecontext of its trace:

1. Open and configure theTrace Explorer page. For informationabout these steps, seeView aggregated trace data.

2. To view details of a span or a trace, do one of the following:

   • Go to the table section of theTrace Explorer page andselect an entry from the tabular section. You can sort the rows of thetables by any table header. You canapply filters torestrict which spans the tables show.

   • Go to the toolbar of theTrace Explorer page,clickpageviewSearch for trace,and enter thetrace ID.

   TheDetails flyout opens and displays a trace and its spans.

   

   The details for the span or trace you selected are shown:

   • TheName column displays the call hierarchy and the first entryincludes the trace ID. This column might also include annotations thatindicate whether generative AI events or log data is available.
   • TheService/workload column displays a service or workload name.Service names are extracted from theOpenTelemetry attributeservice.name, when that attribute is set.If that attribute isn't set and if the service is running onApp Engine, then the App Engine service name isdisplayed. Otherwise, no service is specified.
   • The length of the latency bar is representative of the latency value.
   • The color of the latency bar indicates status. A blue latency bar indicatesa successful completion while a red latency bar indicates an error occurred.
   • A circle on a latency bar indicates a log entry or event is associated withthe span. To change this behavior, use theLogs & events menu.

3. To explore the trace, do the following:

   • Use your pointer to select different spans. When you select a span, thetabs on the page are refreshed to show the span'sattributes, log entries, and events.

   • To search the spans in the trace for keywords,use theFind in Trace field.

     Search applies only to the span name, service or workload name,and attributes fields.You can only search by plain text, and you can't search logs, events, ormetadata.

     For example, if you enterGET, then text in theName,Service/workload, andAttributes panes that displaysGET,ignoring case, is highlighted.

View attributes, log entries, and events

TheDetails flyout for a span contains a tab for each of the following:

• Attributes: This tab lists labels that are attached to spans.For information about labels, seeTrace labels.

• Logs & Events: This tab lists events and log entries whosetrace_id andspan_id fields match the selected span are shown. Onlylog entries that you have permission to view are shown.

  To explore an entry on this tab,clickkeyboard_arrow_downShow more.

  If you prefer to use the Logs Explorer page to view log data, thenclickView logs. The query is set to filter to a specific trace, span,and time range.

  Note: Generative AI events are shown on theInputs/Outputstab. To learn more, seeView generative AI events.

• Stacktraces: Lists the available stacktraces.

• Metadata & Links: Lists general information about the span:

  • Span ID: The span ID is a 64-bit integer other than 0. For details, seeTraceSpan.
  • Parent span ID.
  • Project ID.
  • Start time and end time.

  • Table that lists links to other spans. For each link, this table liststhe following:

    • The key-value pairs for the linked-to span.
    • The trace ID for the linked-to-span. When this field listsCurrent trace, then the span is linking to another span in thesame trace.

    For information about links, see theLinks API reference page.

TheDetails might also include a tab labeledInputs/Outputs tab. This tab displays generative AI events.

View generative AI events

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

To view events related to generative AI agents, use theInputs/Outputs tab. This tab displays events that followtheOpenTelemetry GenAI semantic conventions.

There are two supported configurations:

• Text-based prompts and responses that are attached to the spans asattributes, which are key-value pairs that describe somecharacteristic or an event. For example, the value for the keygen_ai.system identifies the system that provides the generative AIcapabilities. The keys for generative AI attributes begin withgen_ai.

• Multimodal prompts and responses that are stored in aCloud Storage bucket. Cloud Trace identifies these events bycorrelating your trace and log data. To learn about supportedconfigurations, seeCollect and view multimodal prompts and responses.

The following screenshot illustrates how theTrace Explorerpage renders text-based generative AI events that are attached to the spans:

To learn about the sample application that generated the previous screenshot,seeInstrument a LangGraph ReAct Agent with OpenTelemetry.

What's next

• Create and manage trace scopes
• Share traces and spans
• Troubleshoot