Datastore Bulk Delete template [Deprecated]

Beta

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This template isdeprecated and will be removed in Q3 2023. Please migrate to Firestore Bulk Delete template.

The Datastore Bulk Delete template is a pipeline which reads in Entities from Datastore with a given GQL query and then deletes all matching Entities in the selected target project. The pipeline can optionally pass the JSON encoded Datastore Entities to your Javascript UDF, which you can use to filter out Entities by returning null values.

Pipeline requirements

  • Datastore must be set up in the project prior to running the template.
  • If reading and deleting from separate Datastore instances, the DataflowWorker Service Account must have permission to read from one instance and delete from the other.
  • Database writes must be enabled on the Datastore instance.

Template parameters

ParameterDescription
datastoreReadGqlQueryGQL query which specifies which entities to match for deletion. Using a keys-only query may improve performance. For example: "SELECT __key__ FROM MyKind".
datastoreReadProjectIdProject ID of the Datastore instance from which you want to read entities (using your GQL query) that are used for matching.
datastoreDeleteProjectIdProject ID of the Datastore instance from which to delete matching entities. This can be the same asdatastoreReadProjectId if you want to read and delete within the same Datastore instance.
datastoreReadNamespace(Optional) Namespace of requested Entities. Set as "" for default namespace.
datastoreHintNumWorkers(Optional) Hint for the expected number of workers in the Datastore ramp-up throttling step. Default is500.
javascriptTextTransformGcsPath(Optional) The Cloud Storage URI of the.js file that defines the JavaScript user-definedfunction (UDF) you want to use. For example,gs://my-bucket/my-udfs/my_file.js.
javascriptTextTransformFunctionName(Optional) The name of the JavaScript user-defined function (UDF) that you want to use. For example, if your JavaScript function code ismyTransform(inJson) { /*...do stuff...*/ }, then the function name ismyTransform. For sample JavaScript UDFs, seeUDF Examples. If this function returns a value of undefined or null for a given Datastore entity, then that entity is not deleted.

Run the template

Console

  1. Go to the DataflowCreate job from template page.
    2. Go to Create job from template
  2. In theJob name field, enter a unique job name.
  3. Optional: ForRegional endpoint, select a value from the drop-down menu. The default region isus-central1.

    For a list of regions where you can run a Dataflow job, seeDataflow locations.

  4. From theDataflow template drop-down menu, select theBulk Delete Entities in Datastore template.
  5. In the provided parameter fields, enter your parameter values.
  6. ClickRun job.

gcloud

Note: To use the Google Cloud CLI to run classic templates, you must haveGoogle Cloud CLI version 138.0.0 or later.

In your shell or terminal, run the template:

gclouddataflowjobsrunJOB_NAME\--gcs-locationgs://dataflow-templates-REGION_NAME/VERSION/Datastore_to_Datastore_Delete\--regionREGION_NAME\--parameters\datastoreReadGqlQuery="GQL_QUERY",\datastoreReadProjectId=DATASTORE_READ_AND_DELETE_PROJECT_ID,\datastoreDeleteProjectId=DATASTORE_READ_AND_DELETE_PROJECT_ID

Replace the following:

API

To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, seeprojects.templates.launch.

POSThttps://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://dataflow-templates-LOCATION/VERSION/Datastore_to_Datastore_Delete{"jobName":"JOB_NAME","parameters":{"datastoreReadGqlQuery":"GQL_QUERY","datastoreReadProjectId":"DATASTORE_READ_AND_DELETE_PROJECT_ID","datastoreDeleteProjectId":"DATASTORE_READ_AND_DELETE_PROJECT_ID"},"environment":{"zone":"us-central1-f"}}}

Replace the following:

Template source code

Java

/* * Copyright (C) 2018 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); you may not * use this file except in compliance with the License. You may obtain a copy of * the License at * *   http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the * License for the specific language governing permissions and limitations under * the License. */packagecom.google.cloud.teleport.templates;importcom.google.cloud.teleport.metadata.Template;importcom.google.cloud.teleport.metadata.TemplateCategory;importcom.google.cloud.teleport.templates.DatastoreToDatastoreDelete.DatastoreToDatastoreDeleteOptions;importcom.google.cloud.teleport.templates.common.DatastoreConverters.DatastoreDeleteEntityJson;importcom.google.cloud.teleport.templates.common.DatastoreConverters.DatastoreDeleteOptions;importcom.google.cloud.teleport.templates.common.DatastoreConverters.DatastoreReadOptions;importcom.google.cloud.teleport.templates.common.DatastoreConverters.ReadJsonEntities;importcom.google.cloud.teleport.templates.common.FirestoreNestedValueProvider;importcom.google.cloud.teleport.templates.common.JavascriptTextTransformer.JavascriptTextTransformerOptions;importcom.google.cloud.teleport.templates.common.JavascriptTextTransformer.TransformTextViaJavascript;importorg.apache.beam.sdk.Pipeline;importorg.apache.beam.sdk.options.PipelineOptions;importorg.apache.beam.sdk.options.PipelineOptionsFactory;importorg.apache.beam.sdk.options.ValueProvider;/** * Dataflow template which deletes pulled Datastore Entities. * * <p>Check out <a * href="https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/README_Datastore_to_Datastore_Delete.md">README * Datastore</a> or <a * href="https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/README_Firestore_to_Firestore_Delete.md">README * Firestore</a> for instructions on how to use or modify this template. */@Template(name="Datastore_to_Datastore_Delete",category=TemplateCategory.LEGACY,displayName="Bulk Delete Entities in Datastore [Deprecated]",description="A pipeline which reads in Entities (via a GQL query) from Datastore, optionally passes in the JSON encoded "+"Entities to a JavaScript UDF, and then deletes all matching Entities in the selected target project.",optionsClass=DatastoreToDatastoreDeleteOptions.class,skipOptions={"firestoreReadGqlQuery","firestoreReadProjectId","firestoreReadNamespace","firestoreDeleteProjectId","firestoreHintNumWorkers","javascriptTextTransformReloadIntervalMinutes"},documentation="https://cloud.google.com/dataflow/docs/guides/templates/provided/datastore-bulk-delete",contactInformation="https://cloud.google.com/support",requirements={"Datastore must be set up in the project prior to running the template.","If reading and deleting from separate Datastore instances, the Dataflow <a href=\"https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#worker-service-account\">Worker Service Account</a> must have permission to read from one instance and delete from the other."})@Template(name="Firestore_to_Firestore_Delete",category=TemplateCategory.UTILITIES,displayName="Bulk Delete Entities in Firestore (Datastore mode)",description="A pipeline which reads in Entities (via a GQL query) from Firestore, optionally passes in the JSON encoded "+"Entities to a JavaScript UDF, and then deletes all matching Entities in the selected target project.",optionsClass=DatastoreToDatastoreDeleteOptions.class,optionsOrder={DatastoreReadOptions.class,DatastoreDeleteOptions.class,JavascriptTextTransformerOptions.class},skipOptions={"datastoreReadGqlQuery","datastoreReadProjectId","datastoreReadNamespace","datastoreDeleteProjectId","datastoreHintNumWorkers","javascriptTextTransformReloadIntervalMinutes"},documentation="https://cloud.google.com/dataflow/docs/guides/templates/provided/firestore-bulk-delete",contactInformation="https://cloud.google.com/support",requirements={"Firestore must be set up in the project prior to running the template.","If reading and deleting from separate Firestore instances, the Dataflow <a href=\"https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#worker-service-account\">Worker Service Account</a> must have permission to read from one instance and delete from the other."})publicclassDatastoreToDatastoreDelete{publicstatic<T>ValueProvider<T>selectProvidedInput(ValueProvider<T>datastoreInput,ValueProvider<T>firestoreInput){returnnewFirestoreNestedValueProvider<T>(datastoreInput,firestoreInput);}/** Custom PipelineOptions. */publicinterfaceDatastoreToDatastoreDeleteOptionsextendsPipelineOptions,DatastoreReadOptions,JavascriptTextTransformerOptions,DatastoreDeleteOptions{}/**   * Runs a pipeline which reads in Entities from datastore, passes in the JSON encoded Entities to   * a Javascript UDF, and deletes all the Entities.   *   * <p>If the UDF returns value of undefined or null for a given Entity, then that Entity will not   * be deleted.   *   * @param args arguments to the pipeline   */publicstaticvoidmain(String[]args){DatastoreToDatastoreDeleteOptionsoptions=PipelineOptionsFactory.fromArgs(args).withValidation().as(DatastoreToDatastoreDeleteOptions.class);Pipelinepipeline=Pipeline.create(options);pipeline.apply(ReadJsonEntities.newBuilder().setGqlQuery(selectProvidedInput(options.getDatastoreReadGqlQuery(),options.getFirestoreReadGqlQuery())).setProjectId(selectProvidedInput(options.getDatastoreReadProjectId(),options.getFirestoreReadProjectId())).setNamespace(selectProvidedInput(options.getDatastoreReadNamespace(),options.getFirestoreReadNamespace())).build()).apply(TransformTextViaJavascript.newBuilder().setFileSystemPath(options.getJavascriptTextTransformGcsPath()).setFunctionName(options.getJavascriptTextTransformFunctionName()).build()).apply(DatastoreDeleteEntityJson.newBuilder().setProjectId(selectProvidedInput(options.getDatastoreDeleteProjectId(),options.getFirestoreDeleteProjectId())).setHintNumWorkers(selectProvidedInput(options.getDatastoreHintNumWorkers(),options.getFirestoreHintNumWorkers())).build());pipeline.run();}}

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-19 UTC.