After creating a Dataproc cluster, you can adjust ("scale") the clusterby increasing or decreasing the number of primary or secondary worker nodes(horizontal scaling) in the cluster. You can scale aDataproc cluster at any time, even when jobs are running on thecluster. You cannot change the machine type of an existing cluster (verticalscaling). To vertically scale, create a cluster using asupported machine type,then migrate jobs to the new cluster.

Use Dataproc Autoscaling.Instead of manually scaling clusters, enableAutoscalingto have Dataproc set the "right" number ofworkers for your workloads.

You can scale a Dataproc cluster for the following:

1. to increase the number of workers to make a job run faster.
2. to decrease the number of workers to save money (seeGraceful Decommissioningas an option to use when downsizing a cluster to avoid losingwork in progress).
3. to increase the number of nodes to expand available Hadoop Distributed File System (HDFS) storage.

Note: You can't scale asingle-node cluster.

Because clusters can be scaled more than once, you might want toincrease or decrease the cluster size at one time, and then decrease or increasethe size later.

Removing workers: If you remove workersfrom your cluster, make sure that the new cluster size is sufficient to handleyour workload; if it isn't, your jobs may take a long time to complete or maystop responding.

Use scaling

There are three ways you can scale your Dataproc cluster:

1. Use thegcloudcommand-line tool in the gcloud CLI.
2. Edit the cluster configuration in theGoogle Cloud console.
3. Use theREST API.

New workers added to a cluster will use the samemachine typeas existing workers. For example, if a cluster is created withworkers that use then1-standard-8 machine type, new workerswill also use then1-standard-8 machine type.

Note: Dataproc does not support changing worker size(vCPUs, memory, persistent disk) when you add workers to a cluster.

You can scale the number of primary workers or the number of secondary(preemptible) workers, or both. For example, if you only scale the number ofpreemptible workers, the number of primary workers remains the same.

How Dataproc selects cluster nodes for removal

On clusters created with image versions1.5.83+,2.0.57+,and2.1.5+,when scaling down a cluster, Dataproc attempts to minimizethe impact of node removal on running YARN applications by first removinginactive, unhealthy, and idle nodes, then removing nodes withthe fewest running YARN application masters and running containers.

Graceful decommissioning

When youdownscale a cluster, work in progress may stopbefore completion. If you are usingDataproc v 1.2 or later,you can use Graceful Decommissioning, which incorporatesGraceful Decommission of YARN Nodes to finish work in progress on a worker before it is removed from the CloudDataproc cluster.

Note: A worker is not removed from a cluster until running YARN applications arefinished.

Graceful decommissioning and secondary workers

The preemptible (secondary) worker group continues to provision or deleteworkers to reach its expected size even after a cluster scaling operationis marked complete. If you attempt to gracefully decommission a secondary workerand receive an error message similar to the following:

"Secondaryworker group cannot be modified outside of Dataproc. If you recentlycreated or updated this cluster, wait a few minutes before gracefullydecommissioning to allow all secondary instances to join or leave the cluster.Expected secondary worker group size: x, actual size: y",

wait afew minutes then repeat the graceful decommissioning request.

Note:

• You canforcefully decommission preemptible workers at any time.
• You gracefully decommission primary workers at any time

Use graceful decommissioning

Dataproc Graceful Decommissioning incorporatesGraceful Decommission of YARN Nodesto finish work in progress on a worker before it is removed from the CloudDataproc cluster. As a default, graceful decommissioning is disabled. Youenable it by setting a timeout value when you updateyour cluster to remove one or more workers from the cluster.

Cancel a graceful decommissioning scaledown operation

On Dataproc clusters created with image versions2.0.57+or2.1.5+,you can run thegcloud dataproc operations cancelcommand or issue a Dataproc APIoperations.cancelrequest to cancel a graceful decommissioning scaledown operation.

You can obtain the resource name, including the ID, of a scaledown operation byrunninggcloud dataproc operations listor issuing a Dataproc APIoperations.listrequest.

When you cancel a graceful decommissioning scaledown operation:

• workers in aDECOMMISSIONING state are re-commissioned and becomeACTIVEat the completion of the operation's cancellation.

• if the scaledown operation includes label updates, the updatesmay not take effect.

Note: Cancellation of a scaledown operation is asynchronous. Dataprocmakes a best effort to cancel the operation, but success of the cancellation isnot guaranteed.

To verify the status of the cancellation request, you canrun thegcloud dataproc operations describecommand or issue a Dataproc APIoperations.getrequest. If the cancel operation succeeds, the inner operation status is markedasCANCELLED.