gcloud composer environments storage data export - export data from an environment into local storage or Cloud Storage

gcloud composer environments storage data export--destination=DESTINATION(--environment=ENVIRONMENT :--location=LOCATION)[--source=SOURCE][GCLOUD_WIDE_FLAG …]

gs://the-bucket|+--data|||+--file1.txt|+--file2.txt|||+--subdir1|||||+--file3.txt||+--file4.txt

And the local directory '/foo' has the following structure:

/foo|+--file1.txt+--fileX.txt|||+--subdir1|||||+--file3.txt||+--fileY.txt

The following command:

gcloudcomposerenvironmentsstoragedataexportmyenv--destination=/foo

would result in the following structure in the local '/foo' directory:

/foo|+--file1.txt+--file2.txt+--fileX.txt|||+--subdir1|||||+--file3.txt||+--file4.txt||+--fileY.txt

The local files '/foo/file1.txt' and '/foo/subdir1/file3.txt' will beoverwritten with the contents of the corresponding files in the Cloud Storagebucket.

If instead we had run

gcloudcomposerenvironmentsstoragedataexportmyenv--source=subdir1/file3.txt--destination=/foo

the resulting local directory structure would be the following:

/foo|+--file1.txt+--file3.txt+--fileX.txt|||+--subdir1|||||+--file3.txt||+--fileY.txt

No local files would be overwritten since'gs://the-bucket/dags/subdir1/file3.txt' was written to '/foo/file3.txt' insteadof 'foo/subdir1/file3.txt'.

--destination=DESTINATION

The path to an existing local directory or a Cloud Storage bucket/directory intowhich to export files.

Environment resource - The environment from whose Cloud Storage bucket to exportdata.. The arguments in this group can be used to specify the attributes of thisresource. (NOTE) Some attributes are not given arguments in this group but canbe set in other ways.

To set theproject attribute:

• provide the argument--environment on the command line with a fullyspecified name;
• provide the argument--project on the command line;
• set the propertycore/project.

This must be specified.

--environment=ENVIRONMENT

ID of the environment or fully qualified identifier for the environment.

To set theenvironment attribute:

• provide the argument--environment on the command line.

This flag argument must be specified if any of the other arguments in this groupare specified.

--location=LOCATION

Region where Composer environment runs or in which to create the environment.

To set thelocation attribute:

• provide the argument--environment on the command line with a fullyspecified name;
• provide the argument--location on the command line;
• set the propertycomposer/location.

--source=SOURCE

An optional relative path to a file or directory to be exported from the data/subdirectory in the environment's Cloud Storage bucket.

Run$gcloud help for details.