This quickstart shows you how to do the following:

1. Create a Cloud Data Fusion instance.
2. Deploy a sample pipeline that's provided with your Cloud Data Fusioninstance. The pipeline does the following:
   1. Reads a JSON file containing NYT bestseller data fromCloud Storage.
   2. Runs transformations on the file to parse and clean the data.
   3. Loads the top-rated books added in the last week that cost less than $25into BigQuery.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Enable the Cloud Data Fusion API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the API

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Enable the Cloud Data Fusion API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the API

Create a Cloud Data Fusion instance

1. ClickCreate an instance.

   Go to Instances

2. Enter anInstance name.
3. Enter aDescription for your instance.
4. Enter theRegion in which to create the instance.
5. Choose the Cloud Data FusionVersion to use.
6. Choose the Cloud Data FusionEdition.
7. For Cloud Data Fusion versions 6.2.3 and later, in theAuthorization field, choose theDataproc service account to use for running your Cloud Data Fusion pipeline in Dataproc. The default value, Compute Engine account, is pre-selected.
8. ClickCreate. It takes up to 30 minutes for the instance creation process to complete. While Cloud Data Fusion creates your instance, a progress wheel displays next to the instance name on theInstances page. After completion, it turns into a green check mark and indicates that you can start using the instance.

Navigate the Cloud Data Fusion web interface

When using Cloud Data Fusion, you use both the Google Cloud consoleand the separate Cloud Data Fusion web interface.

• In the Google Cloud console, you can do the following:

  • Create a Google Cloud console project
  • Create and delete Cloud Data Fusion instances
  • View the Cloud Data Fusion instance details

• In the Cloud Data Fusion web interface, you can use various pages, suchasStudio orWrangler, to use Cloud Data Fusion functionality.

To navigate the Cloud Data Fusion interface, follow these steps:

1. In the Google Cloud console, open theInstances page.

   Go to Instances

2. In the instanceActions column, click theView Instance link.
3. In the Cloud Data Fusion web interface, use the left navigation panel tonavigate to the page you need.

Deploy a sample pipeline

Sample pipelines are available through the Cloud Data FusionHub,which lets you share reusable Cloud Data Fusion pipelines, plugins,and solutions.

1. In the Cloud Data Fusion web interface, clickHub.
2. In the left panel, clickPipelines.
3. Click theCloud Data Fusion Quickstart pipeline.
4. ClickCreate.
5. In the Cloud Data Fusion Quickstart configuration panel, clickFinish.

6. ClickCustomize Pipeline.

   A visual representation of your pipeline appears on theStudio page,which is a graphical interface for developing data integration pipelines.Available pipeline plugins are listed on the left, and your pipeline isdisplayed on the main canvas area. You can explore your pipeline by holdingthe pointer over each pipelinenode and clickingProperties. Theproperties menu for each node lets you view the objects and operationsassociated with the node.

7. In the top-right menu, clickDeploy. This step submits the pipeline toCloud Data Fusion. You will execute the pipeline in the next section ofthis quickstart.

View your pipeline

The deployed pipeline appears in the pipeline details view, where you can dothe following:

• View the structure and configuration of the pipeline.
• Run the pipeline manually or set up a schedule or a trigger.
• View a summary of historical runs of the pipeline, including executiontimes, logs, and metrics.

Execute your pipeline

In the pipeline details view, clickRun to execute your pipeline.

When executing a pipeline, Cloud Data Fusion does the following:

1. Provisions an ephemeral Dataproc cluster
2. Executes the pipeline on the cluster using Apache Spark
3. Deletes the cluster

View the results

After a few minutes, the pipeline finishes. The pipeline status changes toSucceeded and the number of records processed by each node is displayed.

1. Go to theBigQuery web interface.

2. To view a sample of the results, go to theDataFusionQuickstart datasetin your project, click thetop_rated_inexpensive table, then run a simple query. For example:

   SELECT * FROMPROJECT_ID.GCPQuickStart.top_rated_inexpensive LIMIT 10

   ReplacePROJECT_ID with your project ID.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

1. Delete the BigQuery dataset that your pipeline wrote to in this quickstart.

2. Delete the Cloud Data Fusion instance.

   Note: Deleting your instance does not delete any of your data inthe project.

3. Optional: Delete the project.

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as anappspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Google Cloud console, go to theManage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then clickDelete.
3. In the dialog, type the project ID, and then clickShut down to delete the project.

What's next

• Work through a Cloud Data Fusiontutorial
• Learn about Cloud Data Fusionconcepts