Monitor Cloud Data Fusion system, instance, and pipeline health
This guide explains how to monitor Cloud Data Fusion instances andpipelines. You can view key metrics in the Cloud Data Fusionmonitoring dashboard. For additional aggregation or filtering, you can view themetrics in Cloud Monitoring.
Before you begin
Before you start monitoring Cloud Data Fusion instances and pipelines,review the pricing and grant the required roles.
Pricing
Cloud Monitoring usage incurs charges. For more information, seeGoogle Cloud Observability pricing.
Required roles
To get the permissions that you need to view metrics, ask your administrator to grant you theMonitoring Viewer (roles/monitoring.viewer) IAM role on your project. For more information about granting roles, seeManage access to projects, folders, and organizations.
You might also be able to get the required permissions throughcustom roles or otherpredefined roles.
Monitor in Cloud Data Fusion
View and filter metrics for your instances and pipelines directly inCloud Data Fusion using the monitoring dashboard:
In the Google Cloud console, go to the Cloud Data Fusion page.
ClickInstances, and then click the instance's name to go to theInstance details page.
Click theMonitoring tab. This displays a dashboard with charts for yourinstance and its pipelines.
The dashboard displays the following views:
- Overview: Monitors Cloud Data Fusion systemmetrics.
- Instance: Monitors all Cloud Data Fusion instances in theproject.
- Pipeline: Provides information about pipeline runs and performance.
For detailed information about the metrics in any view, hover over thewidget and clickMore> View in Metrics explorer.
Widgets in the monitoring dashboard
The following sections describe the widgets available in theOverview,Instance, andPipeline views.
Overview view widgets
This table describes the widgets in theOverview view:
| Widget | Description |
|---|---|
| Appfabric service health | Service availability for Appfabric |
| Metric services health | Service availability for metrics |
| Wrangler service health | Service availability for Wrangler |
| Dataset executor health | Service availability for dataset |
| Pipeline studio health | Service availability for Pipeline Studio |
| Runtime service health | Service availability for runtime |
| Log saver service health | Service availability for log saver |
| Metadata service health | Service availability for metadata |
Pipeline view widgets
This table describes the widgets in thePipeline view:
| Widget | Description |
|---|---|
| Successful pipeline runs | Cumulative count of successful pipeline runs. |
| Failed pipeline runs | Cumulative count of failed pipeline runs. |
| Killed pipeline runs | Cumulative count of killed pipeline runs. |
| Rejected pipeline runs | Cumulative count of rejected pipeline runs. |
| Successful pipeline run time | Time taken for successful pipeline runs to complete. |
| Pipeline start latency | Time taken for a pipeline run to reach a "Running" state. |
| Dataproc provisioning latency | Time taken to provision the Dataproc cluster. |
| Dataproc API request count | Cumulative count of API requests made to Dataproc. |
| Successful preview run time | Time taken for successful preview runs to complete. |
| Preview runs | Number of preview runs. |
| Pipeline bytes read | Cumulative count of bytes read by a pipeline. |
| Pipeline bytes written | Cumulative count of bytes written by a pipeline. |
| Pipeline bytes shuffled | Cumulative count of bytes shuffled in a pipeline. |
| Plugin records processed in | Cumulative count of records entering a plugin. |
| Plugin records processed out | Cumulative count of records exiting a plugin. |
Instance view widgets
This table describes the widgets in theInstance view:
| Widget | Description |
|---|---|
| Concurrent pipelines running | Number of pipelines running concurrently. |
| Concurrent pipelines launched | Number of pipelines in a provisioning or starting state. |
| API requests received | Cumulative count of API requests received. |
| API responses count | Cumulative count of API responses sent. |
| Authorization check count | Cumulative count of authorization checks. |
| Authorization check time | Latency of authorization checks. |
| Deployed pipeline count | Number of deployed pipelines. |
| Draft pipeline count | Number of draft pipelines. |
| Namespace count | Number of namespaces. |
View metrics for advanced analysis in Cloud Monitoring
For additional filtering and aggregation options, view your metrics inCloud Monitoring:
In the Google Cloud console, go to the Cloud MonitoringMetrics explorerpage.
Select theCloud Data Fusion monitored resource.
Choose a metric.
Select filters and aggregation.
What's next
- To learn about filters and metrics, see theCloud Data Fusion metrics overview.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-15 UTC.