gcloud alpha dataproc jobs submit flink - submit a Flink job to a cluster

gcloud alpha dataproc jobs submit flink(--class=MAIN_CLASS    |--jar=MAIN_JAR)(--cluster=CLUSTER    |--cluster-labels=[KEY=VALUE,…])[--async][--bucket=BUCKET][--driver-log-levels=[PACKAGE=LEVEL,…]][--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB][--driver-required-vcores=DRIVER_REQUIRED_VCORES][--jars=[JAR,…]][--labels=[KEY=VALUE,…]][--max-failures-per-hour=MAX_FAILURES_PER_HOUR][--max-failures-total=MAX_FAILURES_TOTAL][--properties=[PROPERTY=VALUE,…]][--properties-file=PROPERTIES_FILE][--region=REGION][--savepoint=SAVEPOINT][GCLOUD_WIDE_FLAG …][--JOB_ARGS …]

gcloudalphadataprocjobssubmitflink--cluster=my-cluster--region=us-central1--jar=my_jar.jar--arg1arg2

To submit a Flink job that runs a specific class as an entrypoint:

gcloudalphadataprocjobssubmitflink--cluster=my-cluster--region=us-central1--class=org.my.main.Class--jars=my_jar.jar--arg1arg2

To submit a Flink job that runs a jar that is on the cluster, run:

gcloudalphadataprocjobssubmitflink--cluster=my-cluster--region=us-central1--jar=/usr/lib/flink/examples/streaming/TopSpeedWindowing.jar

[--JOB_ARGS …]

The job arguments to pass.

The '--' argument must be specified between gcloud specific args on the left andJOB_ARGS on the right.

Exactly one of these must be specified:

--class=MAIN_CLASS

The class containing the main method of the driver. Must be in a provided jar orjar that is already on the classpath

--jar=MAIN_JAR

The HCFS URI of jar file containing the driver jar.

Exactly one of these must be specified:

--cluster=CLUSTER

The Dataproc cluster to submit the job to.

--cluster-labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

Labels of Dataproc cluster on which to place the job.

--async

Return immediately, without waiting for the operation in progress to complete.

--bucket=BUCKET

The Cloud Storage bucket to stage files in. Defaults to the cluster's configuredbucket.

--driver-log-levels=[PACKAGE=LEVEL,…]

List of package to log4j log level pairs to configure driver logging. Forexample: root=FATAL,com.example=INFO.

--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB

The memory allocation requested by the job driver in megabytes (MB) forexecution on the driver node group (it is used only by clusters with a drivernode group).

--driver-required-vcores=DRIVER_REQUIRED_VCORES

The vCPU allocation requested by the job driver for execution on the driver nodegroup (it is used only by clusters with a driver node group).

--jars=[JAR,…]

Comma-separated list of jar files to provide to the task manager classpaths.

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

--max-failures-per-hour=MAX_FAILURES_PER_HOUR

Specifies the maximum number of times a job can be restarted per hour in eventof failure. Default is 0 (no retries after job failure).

--max-failures-total=MAX_FAILURES_TOTAL

Specifies the maximum total number of times a job can be restarted after the jobfails. Default is 0 (no retries after job failure).

--properties=[PROPERTY=VALUE,…]

List of key=value pairs to configure Flink. For a list of available properties,see:https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/.

--properties-file=PROPERTIES_FILE

Path to a local file or a file in a Cloud Storage bucket containingconfiguration properties for the job. The client machine running this commandmust have read permission to the file.

Specify properties in the form of property=value in the text file. For example:

# Properties to set for the job:key1=value1key2=value2# Comment out properties not used.# key3=value3

If a property is set in both--properties and--properties-file, the value defined in--propertiestakes precedence.

--region=REGION

Dataproc region to use. Each Dataproc region constitutes an independent resourcenamespace constrained to deploying instances into Compute Engine zones insidethe region. Overrides the defaultdataproc/region property valuefor this command invocation.

--savepoint=SAVEPOINT

HCFS URI of the savepoint that is used to refer to the state of the previouslystopped job. The new job will resume previous state from there.

Run$gcloud help for details.