Cluster Scheduled Deletion
To help avoid incurringGoogle Cloud chargesfor an inactive cluster, use Dataproc's Cluster ScheduledDeletion feature when you create a cluster. This feature provides options todelete a cluster upon the happening of the following events:
- after aspecified cluster idle period
- at a specified future time
- after a specified period that starts from the time of submission of thecluster creation request
Actions that disable scheduled deletion
While a cluster is running, the following actions disablescheduled deletion until the disabling action is reversed:
- Removing IAMDataproc Service Agent roleon theDataproc Service Agent service account
- Disabling the Dataproc API in the cluster project
- EnablingCompute Engine VM deletion protectionon a scheduled deletion cluster VM
- EnablingVPC-Service Controls if theDataprocService Agent service account (Control plane identity)isn't within the perimeter boundary
Calculate cluster idle time
You can use scheduled deletion to delete a cluster after a specified clusteridle time. Idle time is calculated after the cluster is created and clusterprovisioning is complete. The idle time calculation starts when a cluster has norunning jobs.
Thedataproc:dataproc.cluster-ttl.consider-yarn-activitycluster propertyaffects the calculation of cluster idle time, as follows:
- This property is enabled (set to
true) by default. - When this property is enabled, both YARN and Dataproc Jobs APIactivity must be idle to start and continue incrementing the cluster idle timecalculation.
- YARN activity includes pending and running YARN applications.
- Dataproc Jobs API activity includes pending and running jobssubmitted to the Dataproc Jobs API.
- When this property is set to
false, the cluster idletime calculation starts and continues only when DataprocJobs API activity is idle.
Thedataproc:dataproc.cluster-ttl.consider-yarn-activity property appliesto clusters created with image versions released on or after1.4.64,1.5.39,2.0.13, and later image versions. For clusters created with earlierimage versions, only Dataproc Jobs API activity is considered incalculating cluster idle time.
Use cluster scheduled deletion
You can set scheduled deletion values when you create a cluster using theGoogle Cloud CLI, Dataproc API, or Google Cloud console. After youcreate the cluster, you can update the cluster to change or delete scheduleddeletion values previously set on the cluster.
Note: When updating a cluster, you can't combine other cluster update flags withscheduled deletion flagsgcloud CLI
You can create or update scheduled deletion values on a cluster by passing theflags and values listed in the following table to thegcloud dataproc clusters createorgcloud dataproc clusters updatecommands.
| gcloud CLI flag | Description | Value granularity | Min value | Max value |
|---|---|---|---|---|
--delete-max-idle1 | Applies to cluster create and cluster update commands. The duration from the time when the cluster becomes idle after the cluster is created or updated and is in a ready-to-use state to the moment when the cluster starts to delete. Provide the duration inIntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days). Example: "30m": 30 minutes from the moment when the cluster becomes idle. | 1 second | 5 minutes | 14 days |
--no-delete-max-idle | Applies to cluster update command only. Cancels cluster deletion by the previousdelete-max-idle flag setting. | not applicable | not applicable | not applicable |
--delete-expiration-time2 | Applies to cluster create and cluster update commands. The time to start deleting the cluster inISO 8601 datetime format. To generate the datetime in correct format, you can use theTimestamp Generator. For example, "2017-08-22T13:31:48-08:00" specifies an expiration time of 13:21:48 in the UTC -8:00 time zone. | 1 second | 10 minutes from the current time | 14 days from the current time |
--delete-max-age2 | Applies to cluster create and cluster update commands. The duration from the moment of submitting the cluster create request to the moment when the cluster starts to delete. Provide the duration inIntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days). Examples: "30m": 30 minutes from now; "1d": 1 day from now. | 1 second | 10 minutes | 14 days |
--no-delete-max-age | Applies to cluster update command only. Cancels cluster auto-deletion by the previousdelete-max-age ordelete-expiration-time flag setting. | Not applicable | Not applicable | Not applicable |
- You can pass the
delete-max-idleflag with either thedelete-expiration-timeordelete-max-ageflag in yourcluster create or update request. The first to become true takes effect todelete the cluster. - You can pass either thec
delete-expiration-timeflag or thedelete-max-ageflag to the cluster create or update command, but not both.
Cluster creation example:
gcloud dataproc clusters createCLUSTER_NAME \ --region=REGION \ --delete-max-idle=DURATION \ --delete-expiration-time=TIME \ ... other flags ...
Cluster update example:
gcloud dataproc clusters updateCLUSTER_NAME \ --region=REGION \ --delete-max-idle=DURATION \ --no-delete-max-age \ ... other flags
REST API
You can create or update scheduled deletion values on a cluster by setting the DataprocAPIClusterLifecycleConfigfields and values listed in the following table as part of a Dataproccluster.createorcluster.patchAPI request.
| API field | Description | Value granularity | Min value | Max value |
|---|---|---|---|---|
idleDeleteTtl1 | Applies to cluster create and cluster update commands. The duration from the time when the cluster becomes idle after the cluster is created or updated and is in a ready-to-use state to the moment when the cluster starts to delete. When updating a cluster with a new value, the new value must be greater than the previously set value. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". Submit an empty duration to cancel a previously setidleDeleteTtl value. | 1 second | 5 minutes | 14 days |
autoDeleteTime2 | Applies to cluster create and cluster update commands. The time to start deleting the cluster. When updating a cluster with a new time, the new time must be later than the previously set time. When updating, if an empty value is set forautoDeleteTime, it cancels the existing auto delete.Provide a timestamp inRFC 3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z". | 1 second | 10 minutes from the current time | 14 days from the current time |
autoDeleteTtl2 | The duration from the moment of submitting the cluster create or update request to the moment when the cluster starts to delete. When updating a cluster, the new scheduled deletion time (time of the update request plus The new duration) must be later than the previously set cluster deletion time. Submit an empty value to cancel a previously setautoDeleteTtl value. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". | 1 second | 10 minutes | 14 days |
- You can set or update both
idleDeleteTtland eitherautoDeleteTimeorautoDeleteTtlin your cluster create or update request.The first to become true takes effect to delete the cluster. - You can set or update either
autoDeleteTimeorautoDeleteTtlin your request, but not both.
Console
- Open the DataprocCreate a cluster page.
- Select theCustomize cluster panel.
- In theScheduled deletion section, select the options to apply to your cluster.
View Scheduled Deletion cluster settings
gcloud CLI
You can use thegcloud dataproc clusters list command to confirm that a cluster has scheduled deletion enabled.
gcloud dataproc clusters list \ --region=REGION
...NAME WORKER_COUNT ... SCHEDULED_DELETECLUSTER_IDNUMBER ... enabled...
You can use thegcloud dataproc clusters describe command tocheck the clusterLifecycleConfig scheduled deletion settings.
gcloud dataproc clusters describeCLUSTER_NAME \ --region=REGION
...lifecycleConfig: autoDeleteTime: '2018-11-28T19:33:48.146Z' idleDeleteTtl: 1800s idleStartTime: '2018-11-28T18:33:48.146Z'...
TheautoDeleteTime andidleDeleteTtl are thescheduled deletion configuration values set on the cluster.Dataproc generates theidleStartTime value, which isthe latest cluster idle start time. Dataproc deletes thecluster if the cluster remains idle atidleStartTime +idleDeleteTtl.
REST API
You can make aclusters.listrequest to confirm that a cluster has scheduled deletion enabled.
Console
- You can view cluster scheduled deletion settings by selecting the cluster name from the DataprocClusters page in the Google Cloud console.
- From the clusters details page, select theConfiguration tab. Go to the cluster configuration list to view scheduled deletion settings.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-15 UTC.