Horizontal Autoscaling enables Dataflow to choose the appropriatenumber of worker instances for your job, adding or removing workers as needed.Dataflow scales based on the average CPU utilization of theworkers and on the parallelism of a pipeline. The parallelism of a pipeline isan estimate of the number of threads needed to most efficiently process data atany given time.

Horizontal Autoscaling is supported in both batch and streaming pipelines.

Batch autoscaling

Horizontal Autoscaling is enabled by default on all batch pipelines.Dataflow automatically chooses the number of workers based on theestimated total amount of work in each stage of your pipeline. This estimatedepends on the input size and the current throughput. Every 30 seconds,Dataflow re-evaluates the amount of work according to theexecution progress. As the estimated total amount of work increases ordecreases, Dataflow dynamically scales the number of workers up ordown.

The number of workers is sublinear to the amount of work. For example, a jobwith twice the work has fewer than twice the workers.

If any of the following conditions occur, Dataflow eithermaintains or decreases the number of workers, in order to save idle resources:

• The average worker CPU usage is lower than 5%.
• Parallelism is limited due to unparallelizable work, such as un-splittabledata caused by compressed files or I/O modules that don't split.
• The degree of parallelism is fixed, for example when writing to existingfiles in Cloud Storage.

To set an upper bound on the number of workers, set the--maxNumWorkers pipeline option.The default value is2,000.To set a lower bound on the number of workers, set the--dataflow-service-options=min_num_workers service option.These flags are optional.

Streaming autoscaling

For streaming jobs, Horizontal Autoscaling allows Dataflow toadaptively change the number of workers in response to changes in load andresource utilization.

Horizontal Autoscaling is enabled by default for streaming jobs that useStreaming Engine.To enable Horizontal Autoscaling for streaming jobs that don't useStreaming Engine, set the followingpipeline options when you startyour pipeline:

--autoscalingAlgorithm=THROUGHPUT_BASED--maxNumWorkers=MAX_WORKERS

--autoscaling_algorithm=THROUGHPUT_BASED--max_num_workers=MAX_WORKERS

--autoscaling_algorithm=THROUGHPUT_BASED--max_num_workers=MAX_WORKERS

To set a lower bound on the number of workers, set the--dataflow-service-options=min_num_workers service option.When you set this value, horizontal autoscaling doesn't scale below the numberof workers specified. This flag is optional.

While a streaming job is running, you can update the minimum and maximum workersby using anin-flight job update.To adjust the settings, set themin-num-workers andmax-num-workers flags.For more information, seeUpdate the autoscaling range.

Disable Horizontal Autoscaling

To disable Horizontal Autoscaling, set the followingpipeline option when you runthe job.

--autoscalingAlgorithm=NONE

--autoscaling_algorithm=NONE

--autoscaling_algorithm=NONE

Custom sources

If you create a custom data source, you can potentially improve performance byimplementing methods that provide more information to the Horizontal Autoscalingalgorithm:

Limitations

• In jobs running Dataflow Prime, Horizontal Autoscaling is deactivatedduring and up to 10 minutes after Vertical Autoscaling. For more information,seeEffect on Horizontal Autoscaling.
• For pipelines not usingDataflow Shuffle,Dataflow might not be able to scale down the workers effectivelybecause the workers might have shuffled data stored in local disks.
• ThePeriodicImpulsetransform is supported with streaming autoscaling in the Apache Beam SDK versions 2.60.0 and later. If your pipeline usesPeriodicImpulse with an earlier SDK version, then Dataflow workers don't scale down asexpected.

What's next

• Tune Horizontal Autoscaling for streaming pipelines
• Monitor Dataflow autoscaling
• Troubleshoot Dataflow autoscaling