When you deploy anApp Hub application,that application might send telemetry data to multiple Google Cloud projects.To view all of the telemetry generated by your application through a singleinterface, such as through the dashboards thatApplication Monitoring provides, you must configure Google Cloud Observability. This document describesthe required configuration.

This document applies when you deploy your application and then register itwith App Hub, or when deploy your applications by usingtheApplication Design Center.

In addition to configuring Google Cloud Observability, you might want to do the following:

1. Attach application-specific labels to your alerting policies. When youadd these labels, incidents associated with your application areshown on the Application Monitoring dashboards. For information about this step, seeAssociate an alerting policy with an App Hub applicationsection of this document.

2. If you deploy workloads on Google Kubernetes Engine and wantthe traffic, latency, and error rate golden signals displayed on yourApplication Monitoring dashboards, then instrument your application with OpenTelemetry.For more information, seeInstrument for Application Monitoring.

Before you begin

• Identify the project whose observability scope you will configure.This project is one of the following:

  • An App Hub host project.
  • An App Hub management project for your single-project boundary.
  • An App Hub management project for your app-enabled folder.

  In this document, when the type of management project isn't specified,the configuration task applies to all management projects.

• Make sure that you have the necessary Identity and Access Management (IAM) rolesto configure the observability scope. The required IAMroles depend upon whether you plan to create anaggregated sink, which lets youcentralized the storage of log data.

  Configure sink and scopes

  To get the permissions that you need to configure observability scopes and to create an aggregated log sink, ask your administrator to grant you theOrganization Administrator (roles/resourcemanager.organizationAdmin) IAM role on your organization. For more information about granting roles, seeManage access to projects, folders, and organizations.

  You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

  Only configure scopes

  To get the permissions that you need to configure the observability scope, ask your administrator to grant you the following IAM roles:

  • Observability Editor (roles/observability.editor) on yourApp Hub host project or management project
  • Logs Configuration Writer (roles/logging.configWriter) on yourApp Hub host project or management project
  • Monitoring admin (roles/monitoring.admin) on yourApp Hub host project or management project and on each project that you want to add to the metrics scope
  • Cloud Trace User (roles/cloudtrace.user) on yourApp Hub host project or management project
  • App Hub viewer (roles/apphub.viewer) on yourApp Hub host project or management project

  For more information about granting roles, seeManage access to projects, folders, and organizations.

  You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Configure the observability scope

The observability scope controls how explorer and dashboard pages search forthe data to display. Each Google Cloud project contains a singleobservability scope.You don't directly configure a project's observability scope. Instead,for your project, you configure the following:

• The default log scope

  Configure this scope so that when you open theLogs Explorer page or view dashboards, yourapplication's log data is displayed. Make sure this scope lists theprojects and log views which store your application's log data.

• The metrics scope

  Configure this scope so that your charts, for example, those you createby using theMetrics Explorer page, and alerting policies candisplay or monitor your application's metric data. Make sure thisscope lists the projects which store your application's metric data.

• The default trace scope

  Configure this scope so that when you open theTrace Explorer page, your application's trace data is displayed.Make sure this scope lists the projects which store your application'strace data.

The remainder of this section provides guidance about how to configure thesescopes.

Configure and set the default log scope

Do one of the following:

• If you have an organization-levelaggregated sinkthat routes all log data in your organization to a centralized log bucket,then we recommend the following:

  1. Create a log view on the centralizedlog bucket for your application logs.

  2. In yourApp Hub host project or management project,create a log scopeand add your log view, and then set this scope as thedefault log scope.

• If you are using an app-enabled folder and if you don't have anorganization-level aggregated sink or nested folders, then we recommendfollowing:

  1. Create anintercepting aggregated sink for your app-enabled folder,and route those logs to the_Default log bucket of the management project for your app-enabled folder.

  2. In the management project for your app-enabled folder,make sure that the log scope named_Default is set as thedefault log scope.The scope named_Default lists the_AllLogs view on the project's_Default log bucket, which is the centralizedstorage location for your application.

• If you aren't using an aggregated sink, then foryourApp Hub host project or management project, configure thedefault log scopeto list the storage locations of your application's log data.We recommend that you add log views on the log bucketsthat store your log data.

For example, suppose you have configured an app-enabled folder and thatyou aren't using an aggregated sink. Then, you might do the following:

1. In the management project for your app-enabled folder,you create a log scope.

2. You add to the log scope one log view for each project inyour folder, including the management project.

   The view you add is the_AllLogs view on the project's_Defaultlog bucket. This view includes all logs in the_Default log bucket,and the_Default log bucket stores your application log data.

   After you complete this step, the log scope might haveentries like:

   _Default/_AllLogs     my-folder-mp_Default/_AllLogs     project-in-my-folder_Default/_AllLogs     another-project-in-my-folder

3. You save your log scope and set it as thedefault log scope.

Configure the metrics scope

Make sure that the metrics scope foryourApp Hub host project or management projectlists all projects that store your application's metric data:

• For app-enabled folders, Google Cloud Observability attempts tosynchronize the list of projects in your app-enabled folder with the listof projects in the metrics scope. For example, if you add aproject to the app-enabled folder, then a command is issued toadd that project to the metrics scope.

  When the number of projects in your app-enabled folder doesn't exceed yourmetrics scope quota, which defaults to 375projects per metrics scope, then Google Cloud Observability can keep the list ofprojects in the metrics scope synchronized with the list of projects inyour app-enabled folder. For example, suppose the quota is375 projects per metrics scope. Ifyour app-enabled folder contains 100 projects, then the metrics scopelists all projects in your app-enabled folder. If you add projects to yourapp-enabled folder, then they are also added to themetrics scope.

  When the number of projects in your app-enabled folder exceeds yourmetrics scope quota, then the list of projects in the metrics scopewon't include all projects in your app-enabled folder. For example,suppose the quota is 375 projects per metrics scope and suppose yourapp-enabled folder contains 380 projects. After 375 projects are addedto the metrics scope, quota is exhausted and the attempts to add theremaining 5 projects fail. As a result, some application data isn'tavailable to the management project for your app-enabled folder.

  We recommend that you review your usage of your metrics scope quotaand determine whether you need to request a quota update or to manuallymodify the metrics scope. For information about these steps, seeMetrics scopes for app-enabled folders.

• For App Hub host projects and formanagement projects for single-project boundaries, you mustconfigure the metrics scope.

  If you change the set of projects that store your metric data, then you mustalso update your the metrics scope of your host project.

Configure and set the default trace scope

Do the following:

1. For each project that will store your application's trace data, we recommendthat you enable the Observability API:

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

   Enable the API

2. In yourApp Hub host project or management project,create a trace scopeand add projects that store your application's trace data. If you areusing an app-enabled folder, then add the projects in that folder.

3. Set your custom trace scope as thedefault trace scope.

   The topology map can only show trace data from projects in the sameorganization as your host project or your management project.

Associate an alerting policy with an App Hub application

To view your alerting policies from the context of Application Monitoring, you mustassociate them with a service or workload by addingapplication-specific labels to the alerting policy. These user-defined labelsare also included in any incidents created for a policy.To learn more about labels, seeAnnotate incidents with labels.For a list of App Hub labels, seeView application telemetry.

To associate an alerting policy with a workload or service by usingthe Google Cloud console, do the following:

1. In the Google Cloud console, go to thenotifications Alerting page:

   Go toAlerting

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of theGoogle Cloud console, selectyourApp Hub host project or management project.
3. Find the alerting policy, clickmore_vertView more,selectEdit, and then go to theNotifications and name section.
4. In theApplication labels section, select your application andthen select your workload or service.
5. ClickSave policy.

After you complete these steps, labels with the following keys are attached toyour alerting policy. These labels identify your application and your service orworkload:

• apphub_application_location
• apphub_application_id
• apphub_service_id orapphub_workload_id

You can also add user labels to an alerting policy by using theGoogle Cloud CLI, Terraform, or the Cloud Monitoring API.However, you must use the label keys shown in the previous example.For more information, see the following:

• Manage alerting policies by API.
• Manage alerting policies with Terraform.

Grant access

IAM manages access to your log, metric, and tracedata. This section summarizes roles that you might want to grant toprincipals:

• Logs View Accessor(roles/logging.viewAccessor) on the log views listed in the defaultlog scope of the yourApp Hub host project or management project.To learn more about granting access to a log view, seeControl access to a log view.

• Logs Viewer(roles/logging.viewer) on yourApp Hub host project or management projectand on any other projects listed in its default log scope. This rolegrants access to most log entries in the_Default log bucket.For more information, seeLogging roles.

• Monitoring Editor role(roles/monitoring.editor) on yourApp Hub host project or management project.For principals who don't need to create alerting policies, considergranting theMonitoring Viewer role(roles/monitoring.viewer).

• Cloud Trace User(roles/cloudtrace.user) on yourApp Hub host project or management project andon the projects listed in its default trace scope.

• App Hub viewer(roles/apphub.view) on yourApp Hub host project or management project.

What's next

• View application telemetry.

• View application topology.

• Instrument an application for Application Monitoring.

• Application Monitoring supported infrastructure.