Metrics overview
This page describes the metrics that help you monitor the health andperformance of your Cloud Data Fusion instances and pipelines. UseCloud Monitoring to monitor these metrics. The metrics provide insightsinto pipeline runs, instance details, API requests, and authorization checks.
The metrics are categorized as eitherpipeline metrics orinstance metrics:
- Pipeline metrics provide data about individual pipeline runs, such asrun status, duration, latency, and data throughput.
- Instance metrics provide aggregated information about the pipelineswithin an instance, including service availability, the number of deployedpipelines, and API request counts.
Filter and aggregate Cloud Data Fusion pipeline and instance metrics inMonitoring usingmetric andmonitored-resource labels.When youcustomize your metrics views,you can use one or both of these label types.
Cloud Data Fusion Pipeline monitored-resource labels
Filter and aggregate the metrics with the followingCloud Data FusionPipeline monitored-resource labels:
| Label name | Description |
|---|---|
resource_container | The ID of the customer project. |
org_id | The ID of the organization that the customer project belongs to. |
location | The zone or region where the instance is hosted. |
edition | The edition of the Cloud Data Fusion instance. |
is_private_ip_enabled | Whether the instance uses an internal IP address. |
version | The Cloud Data Fusion data plane version of the instance. |
instance_id | The Cloud Data Fusion instance ID. |
namespace | The namespace of the pipeline. |
pipeline_id | The pipeline ID. |
run_id | The run ID for the pipeline. |
Pipeline metric labels
Filter and aggregate the metrics with the following Cloud Data Fusionmetric labels in Monitoring:
| Name | Metric | Description | Metric labels |
|---|---|---|---|
| Pipeline run status | datafusion.googleapis.com/pipeline/v2/runs_completed_count | The cumulative count of pipelines that have completed a run. |
|
| Pipeline run time | datafusion.googleapis.com/pipeline/v2/pipeline_duration | Time taken to complete the pipeline run. |
|
| Pipeline start latency | datafusion.googleapis.com/pipeline/v2/pipeline_start_latency | The time taken for the pipeline to reachRunning state. |
|
| Provisioning latency | datafusion.googleapis.com/pipeline/v2/dataproc/provisioning_latency | The Dataproc cluster provisioning latency. |
|
| Dataproc API requests | datafusion.googleapis.com/pipeline/v2/dataproc/api_request_count | The cumulative count of Dataproc API requests. |
|
| Pipeline preview run time | datafusion.googleapis.com/pipeline/v2/preview_duration | Time taken to complete preview. |
|
| Pipeline bytes written | datafusion.googleapis.com/pipeline/v2/write_bytes_count | The cumulative count of bytes written by a pipeline. |
|
| Pipeline bytes read | datafusion.googleapis.com/pipeline/v2/read_bytes_count | The cumulative count of bytes read by a pipeline. |
|
| Pipeline bytes shuffled | datafusion.googleapis.com/pipeline/v2/shuffle_bytes_count | The cumulative count of bytes shuffled in a pipeline. |
|
| Plugin records processed in | datafusion.googleapis.com/pipeline/v2/plugin/incoming_records_count | Cumulative count of records entering a plugin. |
|
| Plugin records processed out | datafusion.googleapis.com/pipeline/v2/plugin/outgoing_records_count | The cumulative count of records exiting a plugin. |
|
Cloud Data Fusion Instance monitored-resource labels
Filter and aggregate the metrics with the followingCloud Data FusionInstance monitored-resource labels:
| Label name | Description |
|---|---|
resource_container | The ID of the customer project. |
org_id | The ID of the organization that the customer project belongs to. |
location | The zone or region where the instance is hosted. |
edition | The edition of the instance. |
is_private_ip_enabled | Whether the instance uses an internal IP address. |
version | The Cloud Data Fusion data plane version of the instance. |
instance_id | The Cloud Data Fusion instance ID. |
namespace | The namespace name. |
Instance metric labels
Filter and aggregate the metrics with the following Cloud Data Fusionmetric labels in Monitoring:
| Name | Metric | Description | Metric labels |
|---|---|---|---|
| Service status | datafusion.googleapis.com/instance/v2/service_available | The availability of Cloud Data Fusion services. |
|
| Deployed pipeline count | datafusion.googleapis.com/instance/v2/pipelines | The number of deployed pipelines. |
|
| Concurrent pipelines running count | datafusion.googleapis.com/instance/v2/concurrent_pipelines_running | The number of pipelines running concurrently. |
|
| Concurrent pipeline launches count | datafusion.googleapis.com/instance/v2/concurrent_pipelines_launched | The number of pipelines in eitherProvisioning orStarting state. |
|
| CDAP REST API requests received | datafusion.googleapis.com/instance/v2/api_request_count | The cumulative count of REST API requests received by a service in the backend. |
|
| CDAP REST API responses sent | datafusion.googleapis.com/instance/v2/api_response_count | The cumulative count of REST API responses sent by a service in the backend. |
|
| Authorization check count | datafusion.googleapis.com/instance/v2/authorization_check_count | The cumulative count of authorization checks made by the access enforcer. |
|
| Authorization check time | datafusion.googleapis.com/instance/v2/authorization_check_time | The latency of authorization checks made by the access enforcer. |
|
| Draft pipeline count | datafusion.googleapis.com/instance/v2/draft_pipelines | The number of draft pipelines. |
|
| Namespace count | datafusion.googleapis.com/instance/v2/namespaces | The number of namespaces. |
|
What's next
- Learn more aboutviewing Cloud Data Fusion metrics in Monitoring.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-15 UTC.