gcloud storage cp - upload, download, and copy Cloud Storage objects

gcloud storage cp[SOURCE …]DESTINATION[--additional-headers=HEADER=VALUE][--all-versions,-A][--no-clobber,-n][--content-md5=MD5_DIGEST][--continue-on-error,-c][--daisy-chain,-D][--do-not-decompress][--include-managed-folders][--manifest-path=MANIFEST_PATH,-LMANIFEST_PATH][--preserve-posix,-P][--print-created-message,-v][--read-paths-from-stdin,-I][--recursive,-R,-r][--skip-unsupported,-U][--storage-class=STORAGE_CLASS,-sSTORAGE_CLASS][--canned-acl=PREDEFINED_ACL,--predefined-acl=PREDEFINED_ACL,-aPREDEFINED_ACL--[no-]preserve-acl,-p][--gzip-in-flight=[FILE_EXTENSIONS,…],-j [FILE_EXTENSIONS,…]    |--gzip-in-flight-all,-J    |--gzip-local=[FILE_EXTENSIONS,…],-z [FILE_EXTENSIONS,…]    |--gzip-local-all,-Z][--ignore-symlinks    |--preserve-symlinks][--decryption-keys=[DECRYPTION_KEY,…]--encryption-key=ENCRYPTION_KEY][--cache-control=CACHE_CONTROL--content-disposition=CONTENT_DISPOSITION--content-encoding=CONTENT_ENCODING--content-language=CONTENT_LANGUAGE--content-type=CONTENT_TYPE--custom-time=CUSTOM_TIME--clear-custom-metadata    |--custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]    |--remove-custom-metadata=[METADATA_KEYS,…]--update-custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]][--if-generation-match=GENERATION--if-metageneration-match=METAGENERATION][--retain-until=DATETIME--retention-mode=RETENTION_MODE][GCLOUD_WIDE_FLAG …]

Please Note - By default, thecp command does not follow directorysymlinks. You can use the--preserve-symlinks flag to followdirectory symlinks.

gcloudstoragecp*.txtgs://my-bucket

The following command downloads all text files from a bucket to your currentdirectory:

gcloudstoragecpgs://my-bucket/*.txt.

The following command transfers all text files from a bucket to a differentcloud storage provider:

gcloudstoragecpgs://my-bucket/*.txts3://my-bucket

Use the--recursive option to copy an entire directory tree. Thefollowing command uploads the directory treedir:

gcloudstoragecp--recursivedirgs://my-bucket

Recursive listings are similar to adding** to a query, except** matches only cloud objects and will not match prefixes. Forexample, the following would not matchgs://my-bucket/dir/log.txt

gcloudstoragecpgs://my-bucket/**/dirdir

** retrieves a flat list of objects in a single API call. However,** matches folders for non-cloud queries. For example, a folderdir would be copied in the following.

gcloudstoragecp~/Downloads/**/dirgs://my-bucket

[SOURCE …]

The source path(s) to copy.

DESTINATION

The destination path.

--additional-headers=HEADER=VALUE

Includes arbitrary headers in storage API calls. Accepts a comma separated listof key=value pairs, e.g.header1=value1,header2=value2. Overridesthe defaultstorage/additional_headers property value for thiscommand invocation.

--all-versions,-A

Copy all source versions from a source bucket or folder. If not set, only thelive version of each source object is copied.

Note: This option is only useful when the destination bucket has ObjectVersioning enabled. Additionally, the generation numbers of copied versions donot necessarily match the order of the original generation numbers.

--no-clobber,-n

Do not overwrite existing files or objects at the destination. Skipped itemswill be printed. This option may perform an additional GET request for cloudobjects before attempting an upload.

--content-md5=MD5_DIGEST

Manually specified MD5 hash digest for the contents of an uploaded file. Thisflag cannot be used when uploading multiple files. The custom digest is used bythe cloud provider for validation.

--continue-on-error,-c

If any operations are unsuccessful, the command will exit with a non-zero exitstatus after completing the remaining operations. This flag takes effect only insequential execution mode (i.e. processor and thread count are set to 1).Parallelism is default.

--daisy-chain,-D

Copy in "daisy chain" mode, which means copying an object by first downloadingit to the machine where the command is run, then uploading it to the destinationbucket. The default mode is a "copy in the cloud," where data is copied withoutuploading or downloading. During a copy in the cloud, a source composite objectremains composite at its destination. However, you can use daisy chain mode tochange a composite object into a non-composite object. Note: Daisy chain mode isautomatically used when copying between providers.

--do-not-decompress

Do not automatically decompress downloaded gzip files.

--include-managed-folders

Includes managed folders in command operations. For transfers, gcloud storagewill set up managed folders in the destination with the same IAM policy bindingsas the source. Managed folders are only included with recursive cloud-to-cloudtransfers. Please note that for hierarchical namespace buckets, managed foldersare always included. Hence this flag would not be applicable to hierarchicalnamespace buckets.

--manifest-path=MANIFEST_PATH,-LMANIFEST_PATH

Outputs a manifest log file with detailed information about each item that wascopied. This manifest contains the following information for each item:

• Source path.
• Destination path.
• Source size.
• Bytes transferred.
• MD5 hash.
• Transfer start time and date in UTC and ISO 8601 format.
• Transfer completion time and date in UTC and ISO 8601 format.
• Final result of the attempted transfer: OK, error, or skipped.
• Details, if any.

If the manifest file already exists, gcloud storage appends log items to theexisting file.

Objects that are marked as "OK" or "skipped" in the existing manifest file arenot retried by future commands. Objects marked as "error" are retried.

--preserve-posix,-P

Causes POSIX attributes to be preserved when objects are copied. With thisfeature enabled, gcloud storage will copy several fields provided by the statcommand: access time, modification time, owner UID, owner group GID, and themode (permissions) of the file.

For uploads, these attributes are read off of local files and stored in thecloud as custom metadata. For downloads, custom cloud metadata is set as POSIXattributes on files after they are downloaded.

On Windows, this flag will only set and restore access time and modificationtime because Windows doesn't have a notion of POSIX UID, GID, and mode.

--print-created-message,-v

Prints the version-specific URL for each copied object.

--read-paths-from-stdin,-I

Read the list of resources to copy from stdin. No need to enter a sourceargument if this flag is present. Example: "storage cp -Igs://bucket/destination". The input format must consist of one path (e.g.,"Documents/data/file1.txt") or one object URL (e.g.,"gs://example-bucket/event.log") per line. Use a pipe to send the file list tothe command. Example: "cat example-file-list.txt | gcloud storage cp--read-paths-from-stdin gs://example-destination-bucket". Note: To copy thecontents of one file directly from stdin, use "-" as the source argument withoutthe "-I" flag.

--recursive,-R,-r

Recursively copy the contents of any directories that match the source pathexpression.

--skip-unsupported,-U

Skip objects with unsupported object types.

--storage-class=STORAGE_CLASS,-sSTORAGE_CLASS

Specify the storage class of the destination object. If not specified, thedefault storage class of the destination bucket is used. This option is notvalid for copying to non-cloud destinations.

--canned-acl=PREDEFINED_ACL,--predefined-acl=PREDEFINED_ACL,-aPREDEFINED_ACL

Applies predefined, or "canned," ACLs to a resource. See docs for a list ofpredefined ACL constants:https://cloud.google.com/storage/docs/access-control/lists#predefined-acl

--[no-]preserve-acl,-p

Preserves ACLs when copying in the cloud. This option is Cloud Storage-only, andyou need OWNER access to all copied objects. If all objects in the destinationbucket should have the same ACL, you can also set a default object ACL on thatbucket instead of using this flag. Preserving ACLs is the default behavior forupdating existing objects. Use--preserve-acl to enable and--no-preserve-acl to disable.

At most one of these can be specified:

--gzip-in-flight=[FILE_EXTENSIONS,…],-j [FILE_EXTENSIONS,…]

Applies gzip transport encoding to any file upload whose extension matches theinput extension list. This is useful when uploading files with compressiblecontent such as .js, .css, or .html files. This also saves network bandwidthwhile leaving the data uncompressed in Cloud Storage.

When you specify the--gzip-in-flight option, files being uploadedare compressed in-memory and on-the-wire only. Both the local files and CloudStorage objects remain uncompressed. The uploaded objects retain theContent-Type and name of the original files.

--gzip-in-flight-all,-J

Applies gzip transport encoding to file uploads. This option works like the--gzip-in-flight option described above, but it applies to alluploaded files, regardless of extension.

CAUTION: If some of the source files don't compress well, such as binary data,using this option may result in longer uploads.

--gzip-local=[FILE_EXTENSIONS,…],-z [FILE_EXTENSIONS,…]

Applies gzip content encoding to any file upload whose extension matches theinput extension list. This is useful when uploading files with compressiblecontent such as .js, .css, or .html files. This saves network bandwidth andspace in Cloud Storage.

When you specify the--gzip-local option, the data from files iscompressed before it is uploaded, but the original files are left uncompressedon the local disk. The uploaded objects retain theContent-Type andname of the original files. However, theContent-Encoding metadatais set togzip and theCache-Control metadata set tono-transform. The data remains compressed on Cloud Storage serversand will not be decompressed on download by gcloud storage because of theno-transform field.

Since the local gzip option compresses data prior to upload, it is not subjectto the same compression buffer bottleneck of the in-flight gzip option.

--gzip-local-all,-Z

Applies gzip content encoding to file uploads. This option works like the--gzip-local option described above, but it applies to all uploadedfiles, regardless of extension.

CAUTION: If some of the source files don't compress well, such as binary data,using this option may result in files taking up more space in the cloud thanthey would if left uncompressed.

Flags to influence behavior when handling symlinks. Only one value may be set.

At most one of these can be specified:

--ignore-symlinks

Ignore file symlinks instead of copying what they point to.

--preserve-symlinks

Preserve symlinks instead of copying what they point to. With this featureenabled, uploaded symlinks will be represented as placeholders in the cloudwhose content consists of the linked path. Inversely, such placeholders will beconverted to symlinks when downloaded while this feature is enabled, asdescribed athttps://cloud.google.com/storage-transfer/docs/metadata-preservation#posix_to.

Directory symlinks are only followed if this flag is specified.

CAUTION: No validation is applied to the symlink target paths. Once downloaded,preserved symlinks will point to whatever path was specified by the placeholder,regardless of the location or permissions of the path, or whether it actuallyexists.

This feature is not supported on Windows.

--decryption-keys=[DECRYPTION_KEY,…]

A comma-separated list of customer-supplied encryption keys (RFC 4648 section 4base64-encoded AES256 strings) that will be used to decrypt Cloud Storageobjects. Data encrypted with a customer-managed encryption key (CMEK) isdecrypted automatically, so CMEKs do not need to be listed here.

--encryption-key=ENCRYPTION_KEY

The encryption key to use for encrypting target objects. The specifiedencryption key can be a customer-supplied encryption key (An RFC 4648 section 4base64-encoded AES256 string), or a customer-managed encryption key of the formprojects/{project}/locations/{location}/keyRings/{key-ring}/cryptoKeys/{crypto-key}.The specified key also acts as a decryption key, which is useful when copying ormoving encrypted data to a new location. Using this flag in anobjectsupdate command triggers a rewrite of target objects.

--cache-control=CACHE_CONTROL

How caches should handle requests and responses.

--content-disposition=CONTENT_DISPOSITION

How content should be displayed.

--content-encoding=CONTENT_ENCODING

How content is encoded (e.g.gzip).

--content-language=CONTENT_LANGUAGE

Content's language (e.g.en signifies"English").

--content-type=CONTENT_TYPE

Type of data contained in the object (e.g.text/html).

--custom-time=CUSTOM_TIME

Custom time for Cloud Storage objects in RFC 3339 format.

At most one of these can be specified:

--clear-custom-metadata

Clears all custom metadata on objects. When used with--preserve-posix, POSIX attributes will still be stored in custommetadata.

--custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]

Sets custom metadata on objects. When used with--preserve-posix,POSIX attributes are also stored in custom metadata.

Or at least one of these can be specified:

Flags that preserve unspecified existing metadata cannot be used with--custom-metadata or--clear-custom-metadata, but canbe specified together:

--remove-custom-metadata=[METADATA_KEYS,…]

Removes individual custom metadata keys from objects. This flag can be used with--update-custom-metadata. When used with--preserve-posix, POSIX attributes specified by this flag are notpreserved.

--update-custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]

Adds or sets individual custom metadata key value pairs on objects. Existingcustom metadata not specified with this flag is not changed. This flag can beused with--remove-custom-metadata. When keys overlap with thoseprovided by--preserve-posix, values specified by this flag areused.

--if-generation-match=GENERATION

Execute only if the generation matches the generation of the requested object.

--if-metageneration-match=METAGENERATION

Execute only if the metageneration matches the metageneration of the requestedobject.

--retain-until=DATETIME

Ensures the destination object is retained until the specified time in RFC 3339format.

--retention-mode=RETENTION_MODE

Sets the destination object retention mode to either "Locked" or "Unlocked".When retention mode is "Locked", the retain until time can only be increased.RETENTION_MODE must be one of:Locked,Unlocked.

Run$gcloud help for details.