This page describes how to create a Cloud Data Fusion instance with aninternal IP address. You create the instance in aVPC network or aShared VPC network.

A private Cloud Data Fusion instance has the following benefits:

• Connections to the instance are established over aprivate VPC network in your Google Cloud project.Traffic over the network doesn't go through the public internet.

• The instance can connect to your on-premises resources, such as relationaldatabases because your on-premises network connects to theGoogle Cloud private VPC network throughCloud VPN orCloud Interconnect.You can securely access your on-premises resources, such as databases, overthe private network without opening up access to Google Cloud.

Objectives

• Set up the VPC network or the Shared VPC network.
• Allocate an IP range that will be used to deploy the Cloud Data Fusioninstance in the tenant project.
• Create the Cloud Data Fusion private instance.
• Set up the VPC network peering between the VPC thatcontains the Cloud Data Fusion instance and the VPC thatcontains the associated tenant project.
• For Shared VPC networks, set up Identity and Access Management (IAM)permissions.
• If your private instance uses Cloud Data Fusion version 6.2.0 orearlier, create a firewall rule.
• Let different Google Cloud services communicate internally with eachother by enabling Private Google Access on theDataproc subnet.

Before you begin

• To learn about Cloud Data Fusion's deployment architecture, seeNetworking.

Set up the VPC network

If you haven't already done so,create a VPC networkor aShared VPC network.

To set up your VPC network, you must allocate an IP addressrange.

Allocate an IP range

Create a private instance

Create the private Cloud Data Fusion instance in a VPCnetwork or a Shared VPC network.

exportPROJECT=PROJECT_IDexportLOCATION=REGIONexportDATA_FUSION_API_NAME=datafusion.googleapis.com

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" https://$DATA_FUSION_API_NAME/v1/projects/$PROJECT/locations/$LOCATION/instances?instance_id=INSTANCE_ID -X POST -d '{"description": "Private CDF instance created through REST.", "type": "ENTERPRISE", "privateInstance": true, "networkConfig": {"network": "NETWORK_NAME", "ipAllocation": "IP_RANGE"}}'

export PROJECT=PROJECT_IDexport LOCATION=REGIONexport DATA_FUSION_API_NAME=datafusion.googleapis.com

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" https://$DATA_FUSION_API_NAME/v1/projects/$PROJECT/locations/$LOCATION/instances?instanceId=INSTANCE_ID -X POST -d '{"description": "Private CDF instance created through REST.", "type": "ENTERPRISE", "privateInstance": true, "networkConfig": {"network": "projects/SHARED_VPC_HOST_PROJECT_ID/global/networks/NETWORK_NAME", "ipAllocation": "IP_RANGE"}}'

Set up VPC network peering

Cloud Data Fusion services that you use in yourdesign environment(for example: Wrangler, Connection Manager, and Schema Validation) initiatenetwork connections from the tenant project VPC to the sourcesystems. Cloud Data Fusion usesVPC network peering to establish networkconnectivity to the VPC or Shared VPC that contains yourinstance. The VPC network peering lets Cloud Data Fusion accessresources in your network through internal IP addresses using your ownVPC and its controls. To connect with a resource in anothernetwork, see thesteps for connection use cases.

The following section describes how tocreate a peering configurationbetween your network and the Cloud Data Fusiontenant projectnetwork.

Get the tenant project ID

To create a peering configuration, you need thetenant project ID.

1. Go to the Cloud Data FusionInstances page.

   Go to Instances

2. In theInstance Name column, select the instance.

3. On theInstance details page, copy theTenant project ID, which isrequired when you create a peering connection in the following steps.

Create a peering connection

1. Go to theVPC network peering page.

   Go to VPC network peering

2. ClickCreate connection>Continue.

3. On theCreate peering connection page that opens, do the following:

   1. Enter aName for your peering connection.
   2. ForYour VPC network, select the network that contains yourCloud Data Fusion instance.
   3. ForPeered VPC network, selectIn another project.
   4. ForProject ID, enter thetenant project ID you found previously inthis tutorial.

   5. ForVPC network name, select a network or enterINSTANCE_REGION-INSTANCE_ID.

      Replace the following:

      • INSTANCE_REGION: the region in which you created yourCloud Data Fusion instance.
      • INSTANCE_ID: the ID of your Cloud Data Fusion instance.
      Note: When the instance is created, a VPC network namedINSTANCE_REGION-INSTANCE_ID is created in thetenant project. The private Cloud Data Fusion instance is deployedin that VPC. This network already exists with theconfiguration to peer with your customer project VPC.

   6. Select the Internet Protocol version for the peering connection toexchange IPv4 and IPv6 routes between your VPC network andthe peered VPC network. For more information, seeVPC network peering.

   7. SelectExport custom routes so that customroutescan be exported from your VPC network to the tenantVPC network.

   8. Choose whether to allow subnet routes with public IPv4 to be imported orexported into your VPC network.

   9. ClickCreate.

   The VPC network peering becomes active shortly after it is created.

Set up IAM permissions

Create a firewall rule

Create a firewall rule on your VPC network that allows forincoming SSH connections from the IP range you specified when you created yourprivate Cloud Data Fusion instance.

This step is required for Cloud Data Fusion versions earlier than 6.2.0. Itallows communication between Cloud Data Fusion and Dataprocclusters running pipelines.

You can create the firewall rule byusing the Google Cloud consoleor byusing the gcloud CLI.

gcloud compute firewall-rules createFIREWALL_NAME-allow-ssh --allow=tcp:22 --source-ranges=IP_RANGE --network=NETWORK_NAME --project=PROJECT_ID

Steps for connection use cases

The following sections describe connection-related use cases for privateinstances.

Enable Private Google Access

Toaccess resources through internal IP addresses,Cloud Data Fusion must create the Dataproc clusters and runthe data pipelines in a subnet that has Private Google Access. You mustenable Private Google Access for the subnet that contains theDataproc clusters.

• If only one subnet is present in the region where the Dataprocclusters are launched, then the cluster is launched in that subnet.

• If there are multiple subnets in a region, you must configureCloud Data Fusion to select the subnet withPrivate Google Access for launching Dataproc clusters.

  Caution: If Private Google Access isn't enabled on that subnet, the pipeline run fails. To specify the subnet after you create an instance, edit the compute profile.

To enable Private Google Access for the subnet, seePrivate Google Access configuration.

Optional: Connect to other sources

After you create a private instance in Cloud Data Fusion, you can connectto other sources, such as the following use cases:

• On-premises databases and systems running in other VPC networks
• Other Google Cloud services running in their own network in private mode, such asCloud SQL
• Sources on the public internet

Optional: Enable DNS Peering

EnableDNS Peering in thefollowing cases:

• When Cloud Data Fusion connects to systems through hostnames, and not IPaddresses
• When the target system is deployed behind a load balancer, such as it does insome SAP deployments

What's next

• Learn aboutsecurity concepts in Cloud Data Fusion.
• Learn aboutconnecting to resources in external networks.
• Learn about other key Cloud Data Fusionconcepts and features.
• See Cloud Data Fusionpricing.