Run a pipeline against an existing Dataproc cluster

This page describes how to run a pipeline in Cloud Data Fusion againstan existing Dataproc cluster.

By default, Cloud Data Fusion creates ephemeral clusters for each pipeline:it creates a cluster at the beginning of the pipeline run, and then deletes itafter the pipeline run completes. While this behavior saves costs by ensuringthat resources are only created when required, this default behavior might notbe desirable in the following scenarios:

  • If the time it takes to create a new cluster for every pipeline isprohibitive for your use case.

  • If your organization requires cluster creation to be managed centrally; forexample, when you want to enforce certain policies for allDataproc clusters.

For these scenarios, you instead run pipelines against an existing cluster withthe following steps.

Before you begin

You need the following:

Connect to the existing cluster

In Cloud Data Fusion versions 6.2.1 and later, you can connect to anexisting Dataproc cluster when you create a new Compute Engineprofile.

  1. Go to your instance:

    1. In the Google Cloud console, go to the Cloud Data Fusion page.

    2. To open the instance in the Cloud Data Fusion Studio,clickInstances, and then clickView instance.

      Go to Instances

  2. ClickSystem admin.

  3. Click theConfiguration tab.

  4. ClickSystem compute profiles.

  5. ClickCreate new profile. A page of provisioners opens.

  6. ClickExisting Dataproc.

  7. Enter the profile, cluster, and monitoring information.

  8. ClickCreate.

Configure your pipeline to use the custom profile

  1. Go to your instance:

    1. In the Google Cloud console, go to the Cloud Data Fusion page.

    2. To open the instance in the Cloud Data Fusion Studio,clickInstances, and then clickView instance.

      Go to Instances

  2. Go to your pipeline on theStudio page.

  3. ClickConfigure.

  4. ClickCompute config.

  5. Click the profile that you created.

    Use Custom Profile.
    Figure 1: Click the custom profile

  6. Run the pipeline. It runs against the existing Dataproccluster.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.