This section describes Google Cloud options you can use to scheduleworkflows.

Dataproc Workflow Templates

DataprocWorkflow templatesprovide a flexible and easy-to-use mechanism for managing and executing workflows. A Workflow Template is a reusable workflow configuration. It defines a graph of jobs with information on where to run those jobs.

Cloud Scheduler

Cloud Scheduler is a fully managed enterprise-grade cron job scheduler.It allows you to schedule virtually any job, including batch, big data jobs,and Cloud infrastructure operations. It provides simple time-basedscheduling, for example, daily or hourly, without requiring you to write code.

Advantages:

• Enables time-based instantiation of workflow templates based on familiarcron expressions

• No code to write

Tutorial:Workflow using Cloud Scheduler

Cloud Functions

Cloud Run functions is a lightweight compute solution you can use to createsingle-purpose, stand-alone functions that respond to Cloud events withoutthe need to manage a server or runtime environment. You can useCloud Run functions to launch Workflows in response to Pub/Subevents or file changes in Cloud Storage. You can use Cloud Run functionswith Cloud Scheduler for workflows that require the calculation oftime-based parameters.

Advantages:

• Enables workflow instantiation in response to data events, such as new files inCloud Storage or Pub/Sub events.

• Minimal coding required using DataprocGo,Node.js, orPython client libraries

• Dynamically generate workflows and workflow parameters

Tutorial:Workflow using Cloud Run functions

Cloud Composer

Cloud Composer is a managed Apache Airflow service you can useto create, schedule, monitor, and manage workflows.

Advantages:

• Supports time- and event-based scheduling

• Simplified calls to Dataproc using Operators

• Dynamically generate workflows and workflow parameters

• Build data flows that span multiple Google Cloud products

Tutorial:Workflow using Cloud Composer