Use the REST API

To run a template with aREST API request, send an HTTP POST request with your project ID. This request requiresauthorization.

See the REST API reference forprojects.locations.templates.launch to learn more about the available parameters.

Create a custom template batch job

This exampleprojects.locations.templates.launch request creates a batch job from a template that reads a text file and writes an output text file. If the request is successful, the response body contains an instance ofLaunchTemplateResponse.

Modify the following values:

• ReplaceYOUR_PROJECT_ID with your project ID.
• ReplaceLOCATION with the Dataflowregion of your choice.
• ReplaceJOB_NAME with a job name of your choice.
• ReplaceYOUR_BUCKET_NAME with the name of your Cloud Storage bucket.
• SetgcsPath to the Cloud Storage location of the template file.
• Setparameters to your list of key-value pairs.
• SettempLocation to a location where you have write permission. This value is required to run Google-provided templates.

    POST https://dataflow.googleapis.com/v1b3/projects/YOUR_PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://YOUR_BUCKET_NAME/templates/TemplateName    {        "jobName": "JOB_NAME",        "parameters": {            "inputFile" : "gs://YOUR_BUCKET_NAME/input/my_input.txt",            "output": "gs://YOUR_BUCKET_NAME/output/my_output"        },        "environment": {            "tempLocation": "gs://YOUR_BUCKET_NAME/temp",            "zone": "us-central1-f"        }    }

Create a custom template streaming job

This exampleprojects.locations.templates.launch request creates a streaming job from a classic template that reads from a Pub/Sub subscription and writes to a BigQuery table. If you want to launch a Flex Template, useprojects.locations.flexTemplates.launch instead. Theexample template is a Google-provided template. You can modify the path in the template to point to a custom template. The same logic is used to launch Google-provided and custom templates. In this example, the BigQuery table must already exist with the appropriate schema. If successful, the response body contains an instance ofLaunchTemplateResponse.

Modify the following values:

• ReplaceYOUR_PROJECT_ID with your project ID.
• ReplaceLOCATION with the Dataflowregion of your choice.
• ReplaceJOB_NAME with a job name of your choice.
• ReplaceYOUR_BUCKET_NAME with the name of your Cloud Storage bucket.
• ReplaceGCS_PATH with the Cloud Storage location of the template file. The location should start with gs://
• Setparameters to your list of key-value pairs. The parameters listed are specific to this template example. If you're using a custom template, modify the parameters as needed. If you're using the example template, replace the following variables.
  • ReplaceYOUR_SUBSCRIPTION_NAME with your Pub/Sub subscription name.
  • ReplaceYOUR_DATASET with your BigQuery dataset, and replaceYOUR_TABLE_NAME with your BigQuery table name.
• SettempLocation to a location where you have write permission. This value is required to run Google-provided templates.

    POST https://dataflow.googleapis.com/v1b3/projects/YOUR_PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=GCS_PATH    {        "jobName": "JOB_NAME",        "parameters": {            "inputSubscription": "projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION_NAME",            "outputTableSpec": "YOUR_PROJECT_ID:YOUR_DATASET.YOUR_TABLE_NAME"        },        "environment": {            "tempLocation": "gs://YOUR_BUCKET_NAME/temp",            "zone": "us-central1-f"        }    }

Update a custom template streaming job

This exampleprojects.locations.templates.launch request shows you how to update a template streaming job. If you want to update a Flex Template, useprojects.locations.flexTemplates.launch instead.

1. RunExample 2: Creating a custom template streaming job to start a streaming template job.
2. Send the following HTTP POST request, with the following modified values:
   • ReplaceYOUR_PROJECT_ID with your project ID.
   • ReplaceLOCATION with the Dataflowregion of the job that you're updating.
   • ReplaceJOB_NAME with the exact name of the job that you want to update.
   • ReplaceGCS_PATH with the Cloud Storage location of the template file. The location should start with gs://
   • Setparameters to your list of key-value pairs. The parameters listed are specific to this template example. If you're using a custom template, modify the parameters as needed. If you're using the example template, replace the following variables.
     • ReplaceYOUR_SUBSCRIPTION_NAME with your Pub/Sub subscription name.
     • ReplaceYOUR_DATASET with your BigQuery dataset, and replaceYOUR_TABLE_NAME with your BigQuery table name.
   • Use theenvironment parameter to change environment settings, such as the machine type. This example uses the n2-highmem-2 machine type, which has more memory and CPU per worker than the default machine type.

       POST https://dataflow.googleapis.com/v1b3/projects/YOUR_PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=GCS_PATH    {        "jobName": "JOB_NAME",        "parameters": {            "inputSubscription": "projects/YOUR_PROJECT_ID/subscriptions/YOUR_TOPIC_NAME",            "outputTableSpec": "YOUR_PROJECT_ID:YOUR_DATASET.YOUR_TABLE_NAME"        },        "environment": {            "machineType": "n2-highmem-2"        },        "update": true    }

3. Access theDataflow monitoring interface and verify that a new job with the same name was created. This job has the statusUpdated.

Use the Google API Client Libraries

Consider using theGoogle API Client Libraries to easily make calls to the Dataflow REST APIs. This sample script uses theGoogle API Client Library for Python.

In this example, you must set the following variables:

• project: Set to your project ID.
• job: Set to a unique job name of your choice.
• template: Set to the Cloud Storage location of the template file.
• parameters: Set to a dictionary with the template parameters.

To set theregion, include thelocation parameter.

fromgoogleapiclient.discoveryimportbuild# project = 'your-gcp-project'# job = 'unique-job-name'# template = 'gs://dataflow-templates/latest/Word_Count'# parameters = {#     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',#     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',# }dataflow=build("dataflow","v1b3")request=(dataflow.projects().templates().launch(projectId=project,gcsPath=template,body={"jobName":job,"parameters":parameters,},))response=request.execute()

For more information about the available options, see theprojects.locations.templates.launch methodin the Dataflow REST API reference.

Use gcloud CLI

The gcloud CLI can run either a custom or aGoogle-provided template using thegcloud dataflow jobs run command. Examples of running Google-provided templates are documented in theGoogle-provided templates page.

For the following custom template examples, set the following values:

• ReplaceJOB_NAME with a job name of your choice.
• ReplaceYOUR_BUCKET_NAME with the name of your Cloud Storage bucket.
• Set--gcs-location to the Cloud Storage location of the template file.
• Set--parameters to the comma-separated list of parameters to pass to the job. Spaces between commas and values are not allowed.
• To prevent VMs from accepting SSH keys that are stored in project metadata, use theadditional-experiments flag with theblock_project_ssh_keys service option:--additional-experiments=block_project_ssh_keys.

Create a custom template batch job

This example creates a batch job from a template that reads a text file and writes an output text file.

    gcloud dataflow jobs runJOB_NAME \        --gcs-location gs://YOUR_BUCKET_NAME/templates/MyTemplate \        --parameters inputFile=gs://YOUR_BUCKET_NAME/input/my_input.txt,output=gs://YOUR_BUCKET_NAME/output/my_output

The request returns a response with the following format.

    id: 2016-10-11_17_10_59-1234530157620696789    projectId:YOUR_PROJECT_ID    type: JOB_TYPE_BATCH

Create a custom template streaming job

This example creates a streaming job from a template that reads from a Pub/Sub topic and writes to a BigQuery table. The BigQuery table must already exist with the appropriate schema.

    gcloud dataflow jobs runJOB_NAME \        --gcs-location gs://YOUR_BUCKET_NAME/templates/MyTemplate \        --parameters topic=projects/project-identifier/topics/resource-name,table=my_project:my_dataset.my_table_name

The request returns a response with the following format.

    id: 2016-10-11_17_10_59-1234530157620696789    projectId:YOUR_PROJECT_ID    type: JOB_TYPE_STREAMING

For a complete list of flags for thegcloud dataflow jobs run command, see thegcloud CLI reference.

Monitoring and Troubleshooting

TheDataflow monitoring interface lets you to monitor your Dataflow jobs. If a job fails, you can find troubleshooting tips, debugging strategies, and a catalog of common errors in theTroubleshooting Your Pipeline guide.