Storage batch operations is available only if you've configuredStorage Intelligence.

This document describes storage batch operations, aCloud Storage capability that lets you perform operations on billions ofobjects in a serverless manner. Usingstorage batch operations, you can automate large-scale APIoperations on billions of objects, reducing the development time required towrite and maintain scripts for each request.

To learn how to create storage batch operations jobs, seeCreate and manage storage batch operations jobs.

Overview

Storage batch operations let you run one of four transformations onmultiple objects at once: placing an object hold, deleting an object,updating object metadata, and rewriting objects. To usestorage batch operations, you create ajob configuration thatdefines what transformations should be applied to which objects.

Creating a batch operation returns a long-running operation(LRO) that indicates the status of your request: whether the transformation hasbeen applied to all specified objects in your request.

Benefits

• Scalability: Perform transformations on millions of objects with asingle storage batch operations job.
• Serverless execution: Run batch jobs in a serverless environment,eliminating the need to manage infrastructure.
• Automation: Automate complex and repetitive tasks, improving operationalefficiency.
• Reduced development time: Avoid writing and maintaining complex customscripts.
• Performance: Complete time-sensitive operations within the requiredtime. With multiple batch jobs running concurrently on a bucket, you canprocess up to one billion objects within three hours.
• Automatic retries: Automatic retries for failed operations.
• Job monitoring: Detailed progress tracking to monitor the status andcompletion of all jobs.

Use cases

When used withStorage Insights datasets,storage batch operations allow you to accomplish the followingtasks:

• Security management:

  • Set encryption keys on multiple objects using therewrite objectmethod.
  • Apply or remove object holds to control object immutability.

• Compliance:

  • Use object holds to meet data retention requirements for regulatorycompliance.
  • Setobject retention configurationsto manage data lifecycles and compliance policies.
  • Delete data between specific timeframes, to meet wipeout compliancerequirements.

• Data transformation: Perform bulk updates to object metadata.

• Cost optimization: Bulk delete objects in Cloud Storage buckets toreduce storage costs.

Job configurations

Tocreate a storage batch operations job, you'll need to set the following job configurations.Job configurations are parameters that control how the job is defined fordifferent processing requirements.

• Job name: A unique name to identify the storage batch operations job. This is used for tracking, monitoring, and referencing the job. Job names are alphanumeric, for example,job-01.

• Job Description (Optional): A brief description of the job's purpose. This helps with understanding and documenting the job details. For example,Deletes all objects in a bucket.

• Bucket name: The name of the storage bucket containing the objects to be processed. This is essential for locating the input data. For example,my-bucket. You can specify only one bucket name for a job.

• Object selection: The selection criteria that defines which objects to process. You can specify the criteria using any one of the following options:

  • Manifest: Create a manifest and specify its location when you create the storage batch operations job. The manifest is a CSV file, uploaded to Google Cloud, that contains one object or a list of objects that you want to process. Each row in the manifestmust include thebucket andname of the object. You can optionally specifythegeneration of the object. If you don't specify thegeneration, thecurrent version of the object is used.

    The file must include a header row of the following format:

    bucket,name,generation

    The following is an example of the manifest:

    bucket,name,generationbucket_1,object_1,generation_1bucket_1,object_2,generation_2bucket_1,object_3,generation_3

    Caution: Ensure the manifest only includes objects from the bucket provided in the storage batch operations job. Rows referencing other buckets are ignored.

    You can also create a manifest using Storage Insights datasets. For details, seeCreate a manifest using Storage Insights datasets.

  • Object prefixes: Specify a list of prefixes to filter objects within the bucket. Only objects with these prefixes are processed. If empty, all objects in the bucket are processed.

• Job type: Storage batch operations supports the following job types, running a single jobper batch operation.

  • Object deletion: You candelete objects within a bucket. This is crucial for cost optimization, data lifecycle management, and compliance with data deletion policies.

    Caution: By default, Cloud Storage retains soft-deleted objectsfor a duration of seven days. If you have accidentally deleted the objects, youcan restore these soft-deleted objects during this duration. However, if youhave disabledsoft delete for your bucket, you cannot recoverdeleted objects.

  • Metadata updates: You can modify the following object metadata:

    • Custom metadata: You can update any user-defined key-value pairsassociated with the object.

    • Fixed metadata: You can updateCache-Control,Content-Disposition,Content-Encoding,Content-Language,Content-Type,Custom-Time,andRetention configuration. For modifying object holds, use theobject hold job type.

  • Object hold updates: You can enable or disableobject holds.Object holds prevent objects from being deleted or modified, which isessential for compliance and data retention purposes.

  • Object encryption key updates: You can manage thecustomer-managedencryption keys for one or more objects. This includes applying orchanging encryption keys using therewrite object method.

Limitations

Storage batch operations has the following limitations:

• Storage batch operations jobs have a maximum lifetime of 14 days. Anyongoing job that doesn't complete within 14 days of its creation isautomatically cancelled.

• When specifying object prefixes for a job, you can include a maximum of 1000prefixes.

• We don't recommend running more than 20 concurrent batch operations jobs onthe same bucket.

• Storage batch operations is not supported on the followingbuckets:

  • Buckets that haveRequestor Pays enabled.

  • Buckets located in theus-west8 region.

What's next

• Create and manage storage batch operations jobs