gcloud storage rsync - synchronize content of two buckets/directories

gcloud storage rsyncSOURCEDESTINATION[--additional-headers=HEADER=VALUE][--canned-acl=PREDEFINED_ACL,--predefined-acl=PREDEFINED_ACL,-aPREDEFINED_ACL][--checksums-only][--no-clobber,-n][--content-md5=MD5_DIGEST][--continue-on-error,-c][--delete-unmatched-destination-objects][--dry-run][--exclude=[REGEX,…],-x [REGEX,…]][--gzip-in-flight=[FILE_EXTENSIONS,…],-j [FILE_EXTENSIONS,…]][--gzip-in-flight-all,-J][--no-ignore-symlinks][--include-managed-folders][--preserve-posix,-P][--recursive,-R,-r][--skip-if-dest-has-newer-mtime,-u][--skip-unsupported,-U][--decryption-keys=[DECRYPTION_KEY,…]--encryption-key=ENCRYPTION_KEY][--cache-control=CACHE_CONTROL--content-disposition=CONTENT_DISPOSITION--content-encoding=CONTENT_ENCODING--content-language=CONTENT_LANGUAGE--content-type=CONTENT_TYPE--custom-time=CUSTOM_TIME--clear-custom-metadata    |--custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]    |--remove-custom-metadata=[METADATA_KEYS,…]--update-custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]][--if-generation-match=GENERATION--if-metageneration-match=METAGENERATION][GCLOUD_WIDE_FLAG …]

Note, shells (like bash, zsh) sometimes attempt to expand wildcards in ways thatcan be surprising. Also, attempting to copy files whose names contain wildcardcharacters can result in problems.

If synchronizing a large amount of data between clouds you might considersetting up a Google Compute Engine account and runninggcloud storagersync there. Sincegcloud storage rsync cross-provider datatransfers flow through the machine wheregcloud storage rsync isrunning, doing this can make your transfer run significantly faster than on yourlocal workstation.

gcloudstoragersyncdatags://my-bucket/data

To recurse into directories use--recursive:

gcloudstoragersyncdatags://my-bucket/data--recursive

To make the local directorymy-data the same as the contents ofgs://mybucket/data and delete objects in the local directory that are not ings://mybucket/data:

gcloudstoragersyncgs://mybucket/datamy-data--recursive--delete-unmatched-destination-objects

To make the contents of gs://mybucket2 the same as gs://mybucket1 and deleteobjects in gs://mybucket2 that are not in gs://mybucket1:

gcloudstoragersyncgs://mybucket1gs://mybucket2--recursive--delete-unmatched-destination-objects

To copy all objects fromdir1 intodir2 and delete allobjects indir2 which are not indir1:

gcloudstoragersyncdir1dir2--recursive---delete-unmatched-destination-objects

To mirror your objects across cloud providers:

gcloudstoragersyncgs://my-gs-buckets3://my-s3-bucket--recursive--delete-unmatched-destination-objects

To apply gzip compression to only uploaded image files indir:

gcloudstoragersyncdirgs://my-bucket/data--gzip-in-flight=jpeg,jpg,gif,png

To skip the filedir/data1/a.txt:

gcloudstoragersyncdirgs://my-bucket--exclude="data./.*\.txt$"

To skip all .txt and .jpg files:

gcloudstoragersyncdirgs://my-bucket--exclude=".*\.txt$|.*\.jpg$"

SOURCE

The source container path.

DESTINATION

The destination container path.

--additional-headers=HEADER=VALUE

Includes arbitrary headers in storage API calls. Accepts a comma separated listof key=value pairs, e.g.header1=value1,header2=value2. Overridesthe defaultstorage/additional_headers property value for thiscommand invocation.

--canned-acl=PREDEFINED_ACL,--predefined-acl=PREDEFINED_ACL,-aPREDEFINED_ACL

Applies predefined, or "canned," ACLs to a resource. See docs for a list ofpredefined ACL constants:https://cloud.google.com/storage/docs/access-control/lists#predefined-acl

--checksums-only

When comparing objects with matching names at the source and destination, skipmodification time check and compare object hashes. Normally, hashes are onlycompared if modification times are not available.

--no-clobber,-n

Do not overwrite existing files or objects at the destination. Skipped itemswill be printed. This option may perform an additional GET request for cloudobjects before attempting an upload.

--content-md5=MD5_DIGEST

Manually specified MD5 hash digest for the contents of an uploaded file. Thisflag cannot be used when uploading multiple files. The custom digest is used bythe cloud provider for validation.

--continue-on-error,-c

If any operations are unsuccessful, the command will exit with a non-zero exitstatus after completing the remaining operations. This flag takes effect only insequential execution mode (i.e. processor and thread count are set to 1).Parallelism is default.

--delete-unmatched-destination-objects

Delete extra files under DESTINATION not found under SOURCE. By default extrafiles are not deleted. Managed folders are not affected by this flag.

Note: this option can delete data quickly if you specify the wrong source anddestination combination.

--dry-run

Print what operations rsync would perform without actually executing them.

--exclude=[REGEX,…],-x [REGEX,…]

Exclude objects matching regex pattern from rsync.

Note that this is a Python regular expression, not a pure wildcard pattern. Forexample, matching a string ending in "abc" is.*abc$ rather than*abc. Also note that the exclude path is relative, as opposed toabsolute (similar to Linuxrsync andtar excludeoptions).

For the Windows cmd.exe command line interpreter, use^ as anescape character instead of\ and escape the|character. When using Windows PowerShell, use' instead of" and surround the| character with".

--gzip-in-flight=[FILE_EXTENSIONS,…],-j [FILE_EXTENSIONS,…]

Applies gzip transport encoding to any file upload whose extension matches theinput extension list. This is useful when uploading files with compressiblecontent such as .js, .css, or .html files. This also saves network bandwidthwhile leaving the data uncompressed in Cloud Storage.

When you specify the--gzip-in-flight option, files being uploadedare compressed in-memory and on-the-wire only. Both the local files and CloudStorage objects remain uncompressed. The uploaded objects retain theContent-Type and name of the original files.

--gzip-in-flight-all,-J

Applies gzip transport encoding to file uploads. This option works like the--gzip-in-flight option described above, but it applies to alluploaded files, regardless of extension.

CAUTION: If some of the source files don't compress well, such as binary data,using this option may result in longer uploads.

--ignore-symlinks

Ignore file symlinks instead of copying what they point to. Enabled by default,use--no-ignore-symlinks to disable.

--include-managed-folders

Includes managed folders in command operations. For transfers, gcloud storagewill set up managed folders in the destination with the same IAM policy bindingsas the source. Managed folders are only included with recursive cloud-to-cloudtransfers.

--preserve-posix,-P

Causes POSIX attributes to be preserved when objects are copied. With thisfeature enabled, gcloud storage will copy several fields provided by the statcommand: access time, modification time, owner UID, owner group GID, and themode (permissions) of the file.

For uploads, these attributes are read off of local files and stored in thecloud as custom metadata. For downloads, custom cloud metadata is set as POSIXattributes on files after they are downloaded.

On Windows, this flag will only set and restore access time and modificationtime because Windows doesn't have a notion of POSIX UID, GID, and mode.

--recursive,-R,-r

Recursively copy the contents of any directories that match the source pathexpression.

--skip-if-dest-has-newer-mtime,-u

Skip operating on destination object if it has a newer modification time thanthe source.

--skip-unsupported,-U

Skip objects with unsupported object types.

--decryption-keys=[DECRYPTION_KEY,…]

A comma-separated list of customer-supplied encryption keys (RFC 4648 section 4base64-encoded AES256 strings) that will be used to decrypt Cloud Storageobjects. Data encrypted with a customer-managed encryption key (CMEK) isdecrypted automatically, so CMEKs do not need to be listed here.

--encryption-key=ENCRYPTION_KEY

The encryption key to use for encrypting target objects. The specifiedencryption key can be a customer-supplied encryption key (An RFC 4648 section 4base64-encoded AES256 string), or a customer-managed encryption key of the formprojects/{project}/locations/{location}/keyRings/{key-ring}/cryptoKeys/{crypto-key}.The specified key also acts as a decryption key, which is useful when copying ormoving encrypted data to a new location. Using this flag in anobjectsupdate command triggers a rewrite of target objects.

--cache-control=CACHE_CONTROL

How caches should handle requests and responses.

--content-disposition=CONTENT_DISPOSITION

How content should be displayed.

--content-encoding=CONTENT_ENCODING

How content is encoded (e.g.gzip).

--content-language=CONTENT_LANGUAGE

Content's language (e.g.en signifies"English").

--content-type=CONTENT_TYPE

Type of data contained in the object (e.g.text/html).

--custom-time=CUSTOM_TIME

Custom time for Cloud Storage objects in RFC 3339 format.

At most one of these can be specified:

--clear-custom-metadata

Clears all custom metadata on objects. When used with--preserve-posix, POSIX attributes will still be stored in custommetadata.

--custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]

Sets custom metadata on objects. When used with--preserve-posix,POSIX attributes are also stored in custom metadata.

Flags that preserve unspecified existing metadata cannot be used with--custom-metadata or--clear-custom-metadata, but canbe specified together:

--remove-custom-metadata=[METADATA_KEYS,…]

Removes individual custom metadata keys from objects. This flag can be used with--update-custom-metadata. When used with--preserve-posix, POSIX attributes specified by this flag are notpreserved.

--update-custom-metadata=[CUSTOM_METADATA_KEYS_AND_VALUES,…]

Adds or sets individual custom metadata key value pairs on objects. Existingcustom metadata not specified with this flag is not changed. This flag can beused with--remove-custom-metadata. When keys overlap with thoseprovided by--preserve-posix, values specified by this flag areused.

--if-generation-match=GENERATION

Execute only if the generation matches the generation of the requested object.

--if-metageneration-match=METAGENERATION

Execute only if the metageneration matches the metageneration of the requestedobject.

Run$gcloud help for details.