This document describes how to access and view pipeline logs and service logsfor Cloud Data Fusion.

Starting with Cloud Data Fusion version 6.11, pipeline logs andservice logs are available in Cloud Logging.

About log types

Cloud Data Fusion generates several types of logs to help monitor andtroubleshoot data integration processes:

Pricing

Cloud Logging and Cloud Monitoring usage incurs charges. For moreinformation, seeGoogle Cloud Observability pricing.

Optional: Import the Cloud Data Fusion Logging dashboard

To view pipeline logs and service logs using theCloud Data Fusion Logging dashboard, import the dashboard:

1. In the Google Cloud console, go to the Cloud MonitoringDashboards page.

   Go to Dashboards

2. ClickView dashboard templates.

3. Search forCloud Data Fusion Logging and select the dashboard.

4. Clickplaylist_addAdd Cloud Data Fusion Logging dashboard to your list.

View pipeline logs

You can view pipeline logs using the Cloud Data Fusion Loggingdashboard or directly in the Logs Explorer.

View pipeline logs using dashboard

1. If you haven't done already,import theCloud Data Fusion Logging dashboard.

2. In theMy dashboards section, clickCloud Data Fusion Logging.

3. In thePipeline logs section, view the list of pipeline logs. You canfilter the logs by severity, field names, and values.

   To refine your search using queries, use Logs Explorer.

View pipeline logs in Logs Explorer

1. In the Google Cloud console, go to theLogs Explorer page.

   Go to Logs Explorer

2. Enter the following query:

   resource.type="datafusion.googleapis.com/PipelineV2"

   This displays the list of pipeline logs. You can use filters to refinethe results.

Filter pipeline logs

You can filter pipeline logs by run ID, instance ID, pipeline ID, location,namespace, or custom labels.

Every Cloud Data Fusion pipeline run is assigned a uniqueRunID.After you deploy and run your pipeline, you can find theRunID of yourpipeline and view the corresponding pipeline logs.

To filter pipeline logs byRunID, follow these steps:

1. Get the pipeline's RunID.

2. In the Google Cloud console, go to theLogs Explorer page.

3. Enter the following query:

   resource.type="datafusion.googleapis.com/PipelineV2" resource.labels.run_id=RUN_ID

View service logs

You can view service logs using the Cloud Data Fusion Loggingdashboard or in the Logs Explorer.

View service logs using dashboard

1. If you haven't done already,import theCloud Data Fusion Logging dashboard.

2. In theMy dashboards section, clickCloud Data Fusion Logging.

3. In theService logs section, view the list of service logs. You canfilter the logs by severity, field names, and values.

   To refine your search using queries, use Logs Explorer.

View service logs in Logs Explorer

1. In the Google Cloud console, go to theLogs Explorer page.

   Go to Logs Explorer

2. Find the service logs by entering the specific query for that service.

   Service name	Log query
   Appfabric	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="appfabric"
   AppFabric processor	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="appfabric.processor"
   Dataset executor	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="dataset.executor"
   Log saver	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="log.saver"
   Metadata service	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="metadata.service"
   Metrics	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="metrics"
   Pipeline Studio	resource.type="datafusion.googleapis.com/InstanceV2" resource.labels.namespace="system"labels.".userserviceid"="studio"
   Runtime	resource.type="datafusion.googleapis.com/InstanceV2"labels.".serviceId"="runtime"
   Wrangler service	resource.type="datafusion.googleapis.com/InstanceV2"resource.labels.namespace="system"labels.".applicationId"="dataprep"labels.".userserviceid"="service"

Configurable logging in Cloud Data Fusion

Cloud Data Fusion 6.11.0 offers configurablelogging, with Cloud Logging enabled by default. While disablingCloud Logging is possible, it's strongly recommended to keep it enabled toensure you have access to critical pipeline and instance logs.

To disable Cloud Logging, run the following command:

echo '{ "loggingConfig": {"instance_cloud_logging_disabled": "true"}}' | curl -X POST \    -H "Authorization: Bearer $(gcloud auth print-access-token)" \    -H "Content-Type: application/json" \    --data @- \    "https://datafusion.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances?instanceId=$INSTANCE_ID?updateMask=logging_config"

Replace the following:

• PROJECT_ID: the Google Cloud project ID
• LOCATION: thelocation of your instance
• INSTANCE_ID: the ID of your Cloud Data Fusioninstance

What's next

• Learn more aboutCloud Data Fusion audit logging.
• Learn how toview advanced pipeline logs.