To ensure that VM resources are available when your Dataflow jobs needthem, you can use Compute Engine reservations. Reservations provide a highlevel of assurance in obtaining capacity for Compute Engine zonalresources.

To use Compute Engine reservations with Dataflow, perform thefollowing steps:

1. Create a Compute Engine reservation. It can be a single-projectreservation or a shared reservation. For more information, see the followingdocuments:

   • Create a reservation for a single project
   • Create a shared reservation

   The reservation can include GPU or TPU accelerators.

2. When you submit your Dataflow job, pass one of the followingservice options, depending on which version of the Beam SDK you are using:

   • Beam version < 2.29:--experiments=skip_gce_quota_verification
   • Beam version >= 2.29:--dataflow_service_options=automatically_use_created_reservation

To prevent low-priority workloads in the same project from competing forreservations with Dataflow, set the reservation affinity tonone when you create VMs for those workloads. For more information, seeConsuming reserved instances.

In order to use the reservation, the Dataflow workers must matchthe reservation configuration. You might need to set the worker machine type forthe job. For more information, seeWorkers.

Limitations

• All limitations of Compute Engine reservations apply whenDataflow workers consume reservations. SeeHow reservations work.

• Dataflow relies on thedefault consumption orderin Compute Engine. As a result, the following limitations apply:

  • Other workloads in the same project or Organization that don't specify the--reservation flag might compete with Dataflow workloads forproject-specific or shared reservations.

• Dataflow Prime jobs don't consume Compute Engine reservations.

Reservations and accelerators

Dataflow supportsspecifically targetedreservationsfor pipelines using accelerators (GPUs or TPUs). This functionality is generallyavailable with an allowlist. For instructions on using Dataflowaccelerators with specific reservations, contact your account team.

Pricing

Dataflow bills you for VMs fromautomatically consumedreservations while your Dataflow job runs. WhenDataflow isn't using the VMs, Compute Engine bills you.

Compute Engine pricing model

If your Dataflow usage includes VMs fromspecifically targetedreservationsthat have GPUs or TPUs, then compute resources from those reserved VMs arebilled according toCompute EnginePricing. If yourspecifically targeted reservations areattached to aCompute Engine resource-basedcommitment, thenyou also receive applicable resource-based committed use discounts (CUDs) foryour usage. You're also billed a management premium for compute resourcesconsumed in Dataflow. For more pricing details, seeDataflow Pricing.

Dataflow pricing model

For any other type of Compute Engine reservations that you use withDataflow, your usage is billed by using theDataflow pricing model. Dataflowusage from those reservations isn't eligible for resource-based CUDs, even ifthose reservations are attached to a resource-based commitment. This applies tothe following Compute Engine reservations:

• Specifically targeted reservations that don't have GPUs or TPUs
• Allautomatically consumed reservations

What's next

To learn more about Compute Engine reservations, seeReservations of Compute Engine zonal resources.