gcloud beta dataproc jobs submit hadoop - submit a Hadoop job to a cluster

gcloud beta dataproc jobs submit hadoop(--class=MAIN_CLASS    |--jar=MAIN_JAR)(--cluster=CLUSTER    |--cluster-labels=[KEY=VALUE,…])[--archives=[ARCHIVE,…]][--async][--bucket=BUCKET][--driver-log-levels=[PACKAGE=LEVEL,…]][--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB][--driver-required-vcores=DRIVER_REQUIRED_VCORES][--files=[FILE,…]][--jars=[JAR,…]][--labels=[KEY=VALUE,…]][--max-failures-per-hour=MAX_FAILURES_PER_HOUR][--max-failures-total=MAX_FAILURES_TOTAL][--properties=[PROPERTY=VALUE,…]][--properties-file=PROPERTIES_FILE][--region=REGION][GCLOUD_WIDE_FLAG …][--JOB_ARGS …]

gcloudbetadataprocjobssubmithadoop--cluster=my-cluster--jar=my_jar.jar--arg1arg2

To submit a Hadoop job that runs a specific class of a jar, run:

gcloudbetadataprocjobssubmithadoop--cluster=my-cluster--class=org.my.main.Class--jars=my_jar1.jar,my_jar2.jar--arg1arg2

To submit a Hadoop job that runs a jar that is already on the cluster, run:

gcloudbetadataprocjobssubmithadoop--cluster=my-cluster--jar=file:///usr/lib/hadoop-op/hadoop-op-examples.jar--wordcountgs://my_bucket/my_file.txtgs://my_bucket/output

[--JOB_ARGS …]

The arguments to pass to the driver.

The '--' argument must be specified between gcloud specific args on the left andJOB_ARGS on the right.

Exactly one of these must be specified:

--class=MAIN_CLASS

The class containing the main method of the driver. Must be in a provided jar orjar that is already on the classpath

--jar=MAIN_JAR

The HCFS URI of jar file containing the driver jar.

Exactly one of these must be specified:

--cluster=CLUSTER

The Dataproc cluster to submit the job to.

--cluster-labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

Labels of Dataproc cluster on which to place the job.

--archives=[ARCHIVE,…]

Comma separated list of archives to be provided to the job. must be one of thefollowing file formats: .zip, .tar, .tar.gz, or .tgz.

--async

Return immediately, without waiting for the operation in progress to complete.

--bucket=BUCKET

The Cloud Storage bucket to stage files in. Defaults to the cluster's configuredbucket.

--driver-log-levels=[PACKAGE=LEVEL,…]

A list of package to log4j log level pairs to configure driver logging. Forexample: root=FATAL,com.example=INFO

--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB

The memory allocation requested by the job driver in megabytes (MB) forexecution on the driver node group (it is used only by clusters with a drivernode group).

--driver-required-vcores=DRIVER_REQUIRED_VCORES

The vCPU allocation requested by the job driver for execution on the driver nodegroup (it is used only by clusters with a driver node group).

--files=[FILE,…]

Comma separated list of file paths to be provided to the job. A file path caneither be a path to a local file or a path to a file already in a Cloud Storagebucket.

--jars=[JAR,…]

Comma separated list of jar files to be provided to the MR and driverclasspaths.

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

--max-failures-per-hour=MAX_FAILURES_PER_HOUR

Specifies the maximum number of times a job can be restarted per hour in eventof failure. Default is 0 (no retries after job failure).

--max-failures-total=MAX_FAILURES_TOTAL

Specifies the maximum total number of times a job can be restarted after the jobfails. Default is 0 (no retries after job failure).

--properties=[PROPERTY=VALUE,…]

A list of key value pairs to configure Hadoop.

--properties-file=PROPERTIES_FILE

Path to a local file or a file in a Cloud Storage bucket containingconfiguration properties for the job. The client machine running this commandmust have read permission to the file.

Specify properties in the form of property=value in the text file. For example:

# Properties to set for the job:key1=value1key2=value2# Comment out properties not used.# key3=value3

If a property is set in both--properties and--properties-file, the value defined in--propertiestakes precedence.

--region=REGION

Dataproc region to use. Each Dataproc region constitutes an independent resourcenamespace constrained to deploying instances into Compute Engine zones insidethe region. Overrides the defaultdataproc/region property valuefor this command invocation.

Run$gcloud help for details.