Sliced object downloads

One strategy for downloading large files is calledsliced object downloads.In such a download, rangedGET requests are made in parallel, storing datawithin a temporary, pre-allocated destination file. Once all slices havecompleted downloading, the temporary file is renamed to the destination file.

Sliced object downloads can be significantly faster if network and diskspeed are not limiting factors; however, sliced object downloads cause multiplewrites to occur at various locations on disk, so this download strategy candegrade performance for disks with slow seek times, especially when breaking adownload into a large number of slices. Tools such as the Google Cloud CLI havelow default values for the number of slices they create to minimize thepossibility of performance impacts.

Sliced object downloads should always use a fast composable checksum (CRC32C)to verify the data integrity of the slices. To perform sliced object downloads,tools such as the gcloud CLI require a compiled version of crcmod onthe machine performing the download. If compiled crcmod is not available, thegcloud CLI performs non-sliced object downloads instead.

How tools and APIs use sliced object downloads

Depending on how you interact with Cloud Storage, sliced objectdownloads might be managed automatically on your behalf. This section describesthe sliced object download behavior for different tools and provides informationfor how you can modify the behavior.

Console

The Google Cloud console does not perform sliced object downloads.

Command line

By default,gcloud storage cp enables sliced object downloads.You can control how and when the gcloud CLI performs slicedobject downloads by modifying the following properties:

  • storage/sliced_object_download_threshold: The minimum total filesize for performing a sliced object download. You can disable allsliced object downloads by setting this value to0.

  • storage/sliced_object_download_max_components: The maximumnumber of slices to use in the download. Set0 for no limit, inwhich case the number of slices is determined solely bystorage/sliced_object_download_component_size.

  • storage/sliced_object_download_component_size: The target sizefor each download slice. This property is ignored if the total filesize is so large that downloading slices of this size would requiremore slices than allowed, as set instorage/sliced_object_download_max_components.

You can modify these properties by creating anamed configurationand applying the configuration either on a per-command basis by usingthe--configuration project-wide flag or for allgcloud CLI commands by using thegcloud config set command.

No additional local disk space is required when using thegcloud CLI to perform sliced object downloads. If thedownload fails prior to completion, run the command again to resume theslices that failed. Slices that were downloaded successfully before thefailure are not re-downloaded when you retry, except in the case wherethe source object has changed between download attempts.

Temporary downloaded objects appear in the destination directory withthe suffix_.gstmp in their name.

Client libraries

Java

For more information, see theCloud StorageJava API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

You can perform sliced object downloads by settingAllowDivideAndConquer totrue. For example:

importcom.google.cloud.storage.BlobInfo;importcom.google.cloud.storage.transfermanager.DownloadResult;importcom.google.cloud.storage.transfermanager.ParallelDownloadConfig;importcom.google.cloud.storage.transfermanager.TransferManager;importcom.google.cloud.storage.transfermanager.TransferManagerConfig;importjava.nio.file.Path;importjava.util.List;classAllowDivideAndConquerDownload{publicstaticvoiddivideAndConquerDownloadAllowed(List<BlobInfo>blobs,StringbucketName,PathdestinationDirectory){TransferManagertransferManager=TransferManagerConfig.newBuilder().setAllowDivideAndConquerDownload(true).build().getService();ParallelDownloadConfigparallelDownloadConfig=ParallelDownloadConfig.newBuilder().setBucketName(bucketName).setDownloadDirectory(destinationDirectory).build();List<DownloadResult>results=transferManager.downloadBlobs(blobs,parallelDownloadConfig).getDownloadResults();for(DownloadResultresult:results){System.out.println("Download of "+result.getInput().getName()+" completed with status "+result.getStatus());}}}

Node.js

For more information, see theCloud StorageNode.js API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

You can perform sliced object downloads using thedownloadFileInChunks method. For example:

/** * TODO(developer): Uncomment the following lines before running the sample. */// The ID of your GCS bucket// const bucketName = 'your-unique-bucket-name';// The ID of the GCS file to download// const fileName = 'your-file-name';// The path to which the file should be downloaded// const destFileName = '/local/path/to/file.txt';// The size of each chunk to be downloaded// const chunkSize = 1024;// Imports the Google Cloud client libraryconst{Storage,TransferManager}=require('@google-cloud/storage');// Creates a clientconststorage=newStorage();// Creates a transfer manager clientconsttransferManager=newTransferManager(storage.bucket(bucketName));asyncfunctiondownloadFileInChunksWithTransferManager(){// Downloads the filesawaittransferManager.downloadFileInChunks(fileName,{destination:destFileName,chunkSizeBytes:chunkSize,});console.log(`gs://${bucketName}/${fileName} downloaded to${destFileName}.`);}downloadFileInChunksWithTransferManager().catch(console.error);

Python

For more information, see theCloud StoragePython API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

You can perform sliced object downloads using thedownload_chunks_concurrently method. For example:

defdownload_chunks_concurrently(bucket_name,blob_name,filename,chunk_size=32*1024*1024,workers=8):"""Download a single file in chunks, concurrently in a process pool."""# The ID of your GCS bucket# bucket_name = "your-bucket-name"# The file to be downloaded# blob_name = "target-file"# The destination filename or path# filename = ""# The size of each chunk. The performance impact of this value depends on# the use case. The remote service has a minimum of 5 MiB and a maximum of# 5 GiB.# chunk_size = 32 * 1024 * 1024 (32 MiB)# The maximum number of processes to use for the operation. The performance# impact of this value depends on the use case, but smaller files usually# benefit from a higher number of processes. Each additional process occupies# some CPU and memory resources until finished. Threads can be used instead# of processes by passing `worker_type=transfer_manager.THREAD`.# workers=8fromgoogle.cloud.storageimportClient,transfer_managerstorage_client=Client()bucket=storage_client.bucket(bucket_name)blob=bucket.blob(blob_name)transfer_manager.download_chunks_concurrently(blob,filename,chunk_size=chunk_size,max_workers=workers)print("Downloaded{} to{}.".format(blob_name,filename))

REST APIs

Both theJSON API andXML API support rangedGET requests, whichmeans you can use either API to implement your own sliced object downloadstrategy.

In order to protect against data corruption due to the source objectchanging during the download, you should provide thegeneration numberof the source object in each download request for a slice of the object.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.