gcloud beta composer environments storage data import - import data from local storage or Cloud Storage into an environment

gcloud beta composer environments storage data import--source=SOURCE(--environment=ENVIRONMENT :--location=LOCATION)[--destination=DESTINATION][GCLOUD_WIDE_FLAG …]

foo|+--subdir1|||+--file1.txt|+--file2.txt|+--subdir2|||+--file3.txt|+--file4.txt

And the environmentmyenv's Cloud Storage bucket has the followingstructure:

gs://the-bucket|+--data|||+--foo|||||+--subdir1|||||||+--bar.txt

The following command:

gcloudbetacomposerenvironmentsstoragedataimportmyenv--source=/foo

would result in the following structure inmyenv's Cloud Storagebucket:

gs://the-bucket|+--data|||+--foo|||||+--subdir1|||||||+--bar.txt|||+--file1.txt|||+--file2.txt|||||+--subdir2|||||||+--file3.txt|||+--file4.txt

If instead we had run

gcloudbetacomposerenvironmentsstoragedataimportmyenv--source=/foo--destination=bar

the resulting bucket structure would be the following:

gs://the-bucket|+--data|||+--foo|||||+--subdir1|||||||+--bar.txt|||+--bar|||||+--foo|||||||+--subdir1|||||||||+--file1.txt||||+--file2.txt|||||||+--subdir2|||||||||+--file3.txt||||+--file4.txt

--source=SOURCE

Path to a local directory/file or Cloud Storage bucket/object to be importedinto the data/ subdirectory in the environment's Cloud Storage bucket. CloudStorage paths must begin with 'gs://'.

Environment resource - The environment into whose Cloud Storage bucket to importdata.. The arguments in this group can be used to specify the attributes of thisresource. (NOTE) Some attributes are not given arguments in this group but canbe set in other ways.

To set theproject attribute:

• provide the argument--environment on the command line with a fullyspecified name;
• provide the argument--project on the command line;
• set the propertycore/project.

This must be specified.

--environment=ENVIRONMENT

ID of the environment or fully qualified identifier for the environment.

To set theenvironment attribute:

• provide the argument--environment on the command line.

This flag argument must be specified if any of the other arguments in this groupare specified.

--location=LOCATION

Region where Composer environment runs or in which to create the environment.

To set thelocation attribute:

• provide the argument--environment on the command line with a fullyspecified name;
• provide the argument--location on the command line;
• set the propertycomposer/location.

--destination=DESTINATION

An optional subdirectory under the data/ directory in the environment's CloudStorage bucket into which to import files. May contain forward slashes todelimit multiple levels of subdirectory nesting, but should not contain leadingor trailing slashes. If the DESTINATION does not exist, it will be created.

Run$gcloud help for details.