gcloud composer environments create - create and initialize a Cloud Composer environment

gcloud composer environments create(ENVIRONMENT :--location=LOCATION)[--airflow-configs=[KEY=VALUE,…]][--airflow-database-retention-days=AIRFLOW_DATABASE_RETENTION_DAYS][--async][--cloud-sql-machine-type=CLOUD_SQL_MACHINE_TYPE][--cloud-sql-preferred-zone=CLOUD_SQL_PREFERRED_ZONE][--composer-internal-ipv4-cidr-block=COMPOSER_INTERNAL_IPV4_CIDR_BLOCK][--disable-logs-in-cloud-logging-only][--disk-size=DISK_SIZE][--enable-high-resilience][--enable-logs-in-cloud-logging-only][--env-variables=[NAME=VALUE,…]][--environment-size=ENVIRONMENT_SIZE][--labels=[KEY=VALUE,…]][--machine-type=MACHINE_TYPE][--network-attachment=NETWORK_ATTACHMENT][--node-count=NODE_COUNT][--oauth-scopes=[SCOPE,…]][--python-version=PYTHON_VERSION][--service-account=SERVICE_ACCOUNT][--storage-bucket=STORAGE_BUCKET][--support-web-server-plugins][--tags=[TAG,…]][--web-server-machine-type=WEB_SERVER_MACHINE_TYPE][--zone=ZONE][--airflow-version=AIRFLOW_VERSION    |--image-version=IMAGE_VERSION][--cloud-sql-ipv4-cidr=CLOUD_SQL_IPV4_CIDR--composer-network-ipv4-cidr=COMPOSER_NETWORK_IPV4_CIDR--connection-subnetwork=CONNECTION_SUBNETWORK--connection-type=CONNECTION_TYPE--disable-private-environment--enable-private-endpoint--enable-private-environment--enable-privately-used-public-ips--master-ipv4-cidr=MASTER_IPV4_CIDR--web-server-ipv4-cidr=WEB_SERVER_IPV4_CIDR][--cluster-ipv4-cidr=CLUSTER_IPV4_CIDR--cluster-secondary-range-name=CLUSTER_SECONDARY_RANGE_NAME--enable-ip-alias--enable-ip-masq-agent--services-ipv4-cidr=SERVICES_IPV4_CIDR--services-secondary-range-name=SERVICES_SECONDARY_RANGE_NAME][--dag-processor-count=DAG_PROCESSOR_COUNT--dag-processor-cpu=DAG_PROCESSOR_CPU--dag-processor-memory=DAG_PROCESSOR_MEMORY--dag-processor-storage=DAG_PROCESSOR_STORAGE][--disable-cloud-data-lineage-integration    |--enable-cloud-data-lineage-integration][--disable-private-builds-only    |--enable-private-builds-only][--enable-master-authorized-networks--master-authorized-networks=[NETWORK,…]][--enable-scheduled-snapshot-creation--snapshot-creation-schedule=SNAPSHOT_CREATION_SCHEDULE--snapshot-location=SNAPSHOT_LOCATION--snapshot-schedule-timezone=SNAPSHOT_SCHEDULE_TIMEZONE][--enable-triggerer--triggerer-count=TRIGGERER_COUNT--triggerer-cpu=TRIGGERER_CPU--triggerer-memory=TRIGGERER_MEMORY][--kms-key=KMS_KEY :--kms-keyring=KMS_KEYRING--kms-location=KMS_LOCATION--kms-project=KMS_PROJECT][--maintenance-window-end=MAINTENANCE_WINDOW_END--maintenance-window-recurrence=MAINTENANCE_WINDOW_RECURRENCE--maintenance-window-start=MAINTENANCE_WINDOW_START][--max-workers=MAX_WORKERS--min-workers=MIN_WORKERS--scheduler-count=SCHEDULER_COUNT--scheduler-cpu=SCHEDULER_CPU--scheduler-memory=SCHEDULER_MEMORY--scheduler-storage=SCHEDULER_STORAGE--web-server-cpu=WEB_SERVER_CPU--web-server-memory=WEB_SERVER_MEMORY--web-server-storage=WEB_SERVER_STORAGE--worker-cpu=WORKER_CPU--worker-memory=WORKER_MEMORY--worker-storage=WORKER_STORAGE][--network=NETWORK :--subnetwork=SUBNETWORK][--web-server-allow-all    |--web-server-allow-ip=[description=DESCRIPTION],[ip_range=IP_RANGE]    |--web-server-deny-all][GCLOUD_WIDE_FLAG …]

gcloudcomposerenvironmentscreateenv-1

To create a new environment namedenv-1with the Google Compute Engine machine-typen1-standard-8, and the Google ComputeEngine networkmy-network, run:

gcloudcomposerenvironmentscreateenv-1--machine-type=n1-standard-8--network=my-network

Environment resource - The environment to create. The arguments in this groupcan be used to specify the attributes of this resource. (NOTE) Some attributesare not given arguments in this group but can be set in other ways.

To set theproject attribute:

• provide the argumentenvironment on the command line with a fullyspecified name;
• provide the argument--project on the command line;
• set the propertycore/project.

This must be specified.

ENVIRONMENT

ID of the environment or fully qualified identifier for the environment.

To set theenvironment attribute:

• provide the argumentenvironment on the command line.

This positional argument must be specified if any of the other arguments in thisgroup are specified.

--location=LOCATION

Region where Composer environment runs or in which to create the environment.

To set thelocation attribute:

• provide the argumentenvironment on the command line with a fullyspecified name;
• provide the argument--location on the command line;
• set the propertycomposer/location.

--airflow-configs=[KEY=VALUE,…]

A list of Airflow software configuration override KEY=VALUE pairs to set. Forinformation on how to structure KEYs and VALUEs, run$gcloud help composer environmentsupdate.

--airflow-database-retention-days=AIRFLOW_DATABASE_RETENTION_DAYS

The number of days for the Airflow database retention period. If set to 0, theAirflow database retention mechanism will be disabled.

--async

Return immediately, without waiting for the operation in progress to complete.

--cloud-sql-machine-type=CLOUD_SQL_MACHINE_TYPE

Cloud SQL machine type used by the Airflow database. The list of availablemachine types is available here:https://cloud.google.com/composer/pricing#db-machine-types.

--cloud-sql-preferred-zone=CLOUD_SQL_PREFERRED_ZONE

Select cloud sql preferred zone, supported for Composer 2 Environments.

--composer-internal-ipv4-cidr-block=COMPOSER_INTERNAL_IPV4_CIDR_BLOCK

The IP range in CIDR notation to use internally by Cloud Composer. This shouldhave a netmask length of 20. Can be specified for Composer 3 or greater.

--disable-logs-in-cloud-logging-only

Disable logs in cloud logging only, supported for Composer 2 Environments.

--disk-size=DISK_SIZE

The disk size for each VM node in the environment. The minimum size is 20GB, andthe maximum is 64TB. Specified value must be an integer multiple of gigabytes.Cannot be updated after the environment has been created. If units are notprovided, defaults to GB.

--enable-high-resilience

Enable high resilience, supported for Composer 2 Environments.

--enable-logs-in-cloud-logging-only

Enable logs in cloud logging only, supported for Composer 2 Environments.

--env-variables=[NAME=VALUE,…]

A comma-delimited list of environment variableNAME=VALUE pairs toprovide to the Airflow scheduler, worker, and webserver processes. NAME maycontain upper and lowercase letters, digits, and underscores, but they may notbegin with a digit. To include commas as part of aVALUE, seegcloud topicescaping for information about overriding the delimiter.

--environment-size=ENVIRONMENT_SIZE

Size of the environment. Unspecified means that the default option will bechosen.ENVIRONMENT_SIZE must be one of:extra-large,large,medium,small,unspecified.

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

--machine-type=MACHINE_TYPE

The Compute Engine machine type(https://cloud.google.com/compute/docs/machine-types) to use for nodes. Forexample--machine-type=n1-standard-1.

--network-attachment=NETWORK_ATTACHMENT

Cloud Composer Network Attachment, which provides connectivity with a user's VPCnetwork, supported in Composer 3 environments or greater.

--node-count=NODE_COUNT

The number of nodes to create to run the environment.

--oauth-scopes=[SCOPE,…]

The set of Google API scopes to be made available on all of the node VMs.Defaults to ['https://www.googleapis.com/auth/cloud-platform']. Cannot beupdated.

--python-version=PYTHON_VERSION

The Python version to be used within the created environment. Supplied valueshould represent the desired major Python version. Cannot be updated.PYTHON_VERSION must be one of:

2

Created environment will use Python 2

3

Created environment will use Python 3

--service-account=SERVICE_ACCOUNT

The Google Cloud Platform service account to be used by the node VMs. You mustexplicitly specify a service account. Cannot be updated.

--storage-bucket=STORAGE_BUCKET

Name of an exisiting Cloud Storage bucket to be used by the environment.Supported only for Composer 2.4.X and above.

--support-web-server-plugins

Enable the support for web server plugins, supported in Composer 3 or greater.

--tags=[TAG,…]

The set of instance tags applied to all node VMs. Tags are used to identifyvalid sources or targets for network firewalls. Each tag within the list mustcomply with RFC 1035. Cannot be updated.

--web-server-machine-type=WEB_SERVER_MACHINE_TYPE

machine type used by the Airflow web server. The list of available machine typesis available here:https://cloud.google.com/composer/pricing.

--zone=ZONE

The Compute Engine zone in which the environment will be created. For example--zone=us-central1-a.

At most one of these can be specified:

--airflow-version=AIRFLOW_VERSION

Version of Apache Airflow to run in the environment.

Mustbeoftheform`X[.Y[.Z]]`,where`[]`denotesoptionalfragments.Examples:`2`,`2.3`,`2.3.4`.

ThecurrentCloudComposerversionwillbeusedwithinthecreatedenvironment.TheApacheAirflowversionisasemanticversionoranaliasintheformofmajorormajor.minorversionnumbers,resolvedtothelatestmatchingApacheAirflowversionsupportedinthecurrentCloudComposerversion.Theresolvedversionisstoredinthecreatedenvironment.

--image-version=IMAGE_VERSION

Version of the image to run in the environment.

TheimageversionencapsulatestheversionsofbothCloudComposerandApacheAirflow.Mustbeoftheform`composer-A[.B.C[-D.E]]-airflow-X[.Y[.Z]]`,where`[]`denotesoptionalfragments.

Examples:`composer-2-airflow-2`,`composer-2-airflow-2.2`,`composer-2.1.2-airflow-2.3.4`.

TheCloudComposerportionoftheimageversionisasemanticversionoranaliasintheformofmajorversionnumberor`latest`,resolvedtothecurrentCloudComposerversion.TheApacheAirflowportionoftheimageversionisasemanticversionoranaliasintheformofmajorormajor.minorversionnumbers,resolvedtothelatestmatchingApacheAirflowversionsupportedinthegivenCloudComposerversion.Theresolvedversionsarestoredinthecreatedenvironment.

Private Clusters

--cloud-sql-ipv4-cidr=CLOUD_SQL_IPV4_CIDR

IPv4 CIDR range to use for the Cloud SQL network. This should have a size of thenetmask not greater than 24.

Cannot be specified unless--enable-private-environment is alsospecified.

--composer-network-ipv4-cidr=COMPOSER_NETWORK_IPV4_CIDR

IPv4 CIDR range to use for the Composer network. This must have a size of thenetmask between 24 and 29.

Can be specified for Composer 2.X or greater. Cannot be specified unless--enable-private-environment is also specified.

--connection-subnetwork=CONNECTION_SUBNETWORK

Subnetwork from which an IP address for internal communications will bereserved. Needs to belong to the Compute network to which the environment isconnected. Can be the same subnetwork as the one to which the environment isconnected.

Can be specified for Composer 2.X or greater. Cannot be specified unless--enable-private-environment is also specified.

--connection-type=CONNECTION_TYPE

Mode of internal communication within the Composer environment. Must be one ofVPC_PEERING orPRIVATE_SERVICE_CONNECT.

Can be specified for Composer 2.X or greater. Cannot be specified unless--enable-private-environment is also specified. Cannot be set toVPC_PEERING if--connection-subnetwork is alsospecified.

CONNECTION_TYPE must be one of:connection-type-unspecified,private-service-connect,vpc-peering.

--disable-private-environment

Enables internet connection for Composer components.

When used with Composer 2, this means the environment cluster is created withpublic IP addresses on the cluster nodes.

--enable-private-endpoint

Environment cluster is managed using the private IP address of the master APIendpoint. Therefore access to the master endpoint must be from internal IPaddresses.

If not specified, the master API endpoint will be accessible by its public IPaddress.

Cannot be specified unless--enable-private-environment is alsospecified.

--enable-private-environment

Disables internet connection from any Composer component.

When used with Composer 2, the environment cluster is created with no public IPaddresses on the cluster nodes. If not specified, cluster nodes will be assignedpublic IP addresses.

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

--enable-privately-used-public-ips

When enabled GKE pods and services may use public(non-RFC1918) IP rangesprivately. The ranges are specified by '--cluster-ipv4-cidr' and--services-ipv4-cidr flags.

Cannot be specified unless--enable-private-environment is alsospecified.

--master-ipv4-cidr=MASTER_IPV4_CIDR

IPv4 CIDR range to use for the cluster master network. This should have a sizeof the netmask between 23 and 28.

Cannot be specified unless--enable-private-environment is alsospecified.

--web-server-ipv4-cidr=WEB_SERVER_IPV4_CIDR

IPv4 CIDR range to use for the Airflow web server network. This should have asize of the netmask between 24 and 29.

Cannot be specified unless--enable-private-environment is alsospecified.

IP Alias (VPC-native)

--cluster-ipv4-cidr=CLUSTER_IPV4_CIDR

IP address range for the pods in this cluster in CIDR notation (e.g.10.0.0.0/14).

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

--cluster-secondary-range-name=CLUSTER_SECONDARY_RANGE_NAME

Secondary range to be used as the source for pod IPs. Alias ranges will beallocated from this secondary range. NAME must be the name of an existingsecondary range in the cluster subnetwork.

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

--enable-ip-alias

Enable use of alias IPs (https://cloud.google.com/compute/docs/alias-ip/) forPod IPs. This will require at least two secondary ranges in the subnetwork, onefor the pod IPs and another to reserve space for the services range.

--enable-ip-masq-agent

When enabled, the IP Masquarade Agent(https://cloud.google.com/composer/docs/enable-ip-masquerade-agent) is deployedto your environment's cluster. It performs many-to-one IP address translationsto hide a pod's IP address behind the cluster node's IP address. This is donewhen sending traffic to destinations outside the cluster's pod CIDR range.

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

--services-ipv4-cidr=SERVICES_IPV4_CIDR

IP range for the services IPs.

Can be specified as a netmask size (e.g. '/20') or as in CIDR notion (e.g.'10.100.0.0/20'). If given as a netmask size, the IP range will be chosenautomatically from the available space in the network.

If unspecified, the services CIDR range will be chosen with a default mask size.

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

--services-secondary-range-name=SERVICES_SECONDARY_RANGE_NAME

Secondary range to be used for services (e.g. ClusterIPs). NAME must be the nameof an existing secondary range in the cluster subnetwork.

When used with Composer 1.x, cannot be specified unless--enable-ip-alias is also specified.

Group of arguments for setting dag processor settings in Composer 3 or greater.

--dag-processor-count=DAG_PROCESSOR_COUNT

Number of dag processors, supported in Composer 3 environments or greater.

--dag-processor-cpu=DAG_PROCESSOR_CPU

CPU allocated to Airflow dag processor, supported in Composer 3 environments orgreater.

--dag-processor-memory=DAG_PROCESSOR_MEMORY

Memory allocated to Airflow dag processor, ex. 1GB, 3GB, 2. If units are notprovided, defaults to GB, supported in Composer 3 environments or greater.

--dag-processor-storage=DAG_PROCESSOR_STORAGE

Storage allocated to Airflow dag processor, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB, supported in Composer 3 environments or greater.

At most one of these can be specified:

--disable-cloud-data-lineage-integration

Disable Cloud Data Lineage integration, supported for Composer 2 Environments.

--enable-cloud-data-lineage-integration

Enable Cloud Data Lineage integration, supported for Composer 2 Environments.

At most one of these can be specified:

--disable-private-builds-only

Builds performed during operations that install Python packages have an accessto the internet, supported in Composer 3 or greater.

--enable-private-builds-only

Builds performed during operations that install Python packages have onlyprivate connectivity to Google services, supported in Composer 3 or greater.

Master Authorized Networks configuration

--enable-master-authorized-networks

Enable Master Authorized Networks feature(https://cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks/) inthe Composer Environment's GKE cluster.

--master-authorized-networks=[NETWORK,…]

Comma separated Master Authorized Networks specified in CIDR notation.

Cannot be specified unless--enable-master-authorized-networks isalso specified.

Group of arguments for setting scheduled snapshots settings in Composer 2.0.32or greater.

--enable-scheduled-snapshot-creation

When specified, snapshots of the environment will be created according to aschedule. Can be specified for Composer 2.0.32 or greater.

This flag argument must be specified if any of the other arguments in this groupare specified.

--snapshot-creation-schedule=SNAPSHOT_CREATION_SCHEDULE

Cron expression specifying when snapshots of the environment should be created.Can be specified for Composer 2.0.32 or greater.

This flag argument must be specified if any of the other arguments in this groupare specified.

--snapshot-location=SNAPSHOT_LOCATION

The Cloud Storage location for storing automatically created snapshots. Can bespecified for Composer 2.0.32 or greater.

This flag argument must be specified if any of the other arguments in this groupare specified.

--snapshot-schedule-timezone=SNAPSHOT_SCHEDULE_TIMEZONE

Timezone that sets the context to interpret snapshot_creation_schedule. Can bespecified for Composer 2.0.32 or greater.

This flag argument must be specified if any of the other arguments in this groupare specified.

Group of arguments for setting triggerer settings in Composer 2.0.31 or greater.

--enable-triggerer

(DEPRECATED) Enable use of a triggerer, supported in the Environments withComposer 2.0.31 and Airflow 2.2.5 and greater.

This flag is deprecated. Use --triggerer-count instead.

--triggerer-count=TRIGGERER_COUNT

Number of triggerers, supported in the Environments with Composer 2.0.31 andAirflow 2.2.5 and greater.

--triggerer-cpu=TRIGGERER_CPU

CPU allocated to Airflow triggerer. Supported in the Environments with Composer2.0.31 and Airflow 2.2.5 and greater.

--triggerer-memory=TRIGGERER_MEMORY

Memory allocated to Airflow triggerer, ex. 512MB, 3GB, 2. If units are notprovided, defaults to GB. Supported in the Environments with Composer 2.0.31 andAirflow 2.2.5 and greater.

Key resource - The Cloud KMS (Key Management Service) cryptokey that will beused to protect the environment. The 'Cloud Composer Service Agent' serviceaccount must hold permission 'Cloud KMS CryptoKey Encrypter/Decrypter'. Thearguments in this group can be used to specify the attributes of this resource.

--kms-key=KMS_KEY

ID of the key or fully qualified identifier for the key.

To set thekms-key attribute:

• provide the argument--kms-key on the command line.

This flag argument must be specified if any of the other arguments in this groupare specified.

--kms-keyring=KMS_KEYRING

The KMS keyring of the key.

To set thekms-keyring attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-keyring on the command line.

--kms-location=KMS_LOCATION

The Google Cloud location for the key.

To set thekms-location attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-location on the command line.

--kms-project=KMS_PROJECT

The Google Cloud project for the key.

To set thekms-project attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-project on the command line;
• set the propertycore/project.

Group of arguments for setting the maintenance window value.

--maintenance-window-end=MAINTENANCE_WINDOW_END

End time of the mantenance window in the form of the full date. Only the time ofthe day is used as a reference for an ending time of the window with a providedrecurrence. Specified date must take place after the one specified as a startdate, the difference between will be used as a length of a single maintenancewindow. See $gcloud topicdatetimes for information on time formats.

This flag argument must be specified if any of the other arguments in this groupare specified.

--maintenance-window-recurrence=MAINTENANCE_WINDOW_RECURRENCE

An RFC 5545 RRULE, specifying how the maintenance window will recur. The minimumrequirement for the length of the maintenance window is 12 hours a week. OnlyFREQ=DAILY and FREQ=WEEKLY rules are supported.

This flag argument must be specified if any of the other arguments in this groupare specified.

--maintenance-window-start=MAINTENANCE_WINDOW_START

Start time of the mantenance window in the form of the full date. Only the timeof the day is used as a reference for a starting time of the window with aprovided recurrence. See $gcloud topic datetimes forinformation on time formats.

This flag argument must be specified if any of the other arguments in this groupare specified.

Group of arguments for setting workloads configuration in Composer 2.X orgreater (--scheduler-count flag is available for Composer 1.X as well).

--max-workers=MAX_WORKERS

Maximum number of workers in the Environment.

--min-workers=MIN_WORKERS

Minimum number of workers in the Environment.

--scheduler-count=SCHEDULER_COUNT

Number of schedulers, supported in the Environments with Airflow 2.0.1 andlater.

--scheduler-cpu=SCHEDULER_CPU

CPU allocated to Airflow scheduler.

--scheduler-memory=SCHEDULER_MEMORY

Memory allocated to Airflow scheduler, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

--scheduler-storage=SCHEDULER_STORAGE

Storage allocated to Airflow scheduler, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

--web-server-cpu=WEB_SERVER_CPU

CPU allocated to each Airflow web server

--web-server-memory=WEB_SERVER_MEMORY

Memory allocated to Airflow web server, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

--web-server-storage=WEB_SERVER_STORAGE

Storage allocated to Airflow web server, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

--worker-cpu=WORKER_CPU

CPU allocated to each Airflow worker

--worker-memory=WORKER_MEMORY

Memory allocated to Airflow worker, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

--worker-storage=WORKER_STORAGE

Storage allocated to Airflow worker, ex. 600MB, 3GB, 2. If units are notprovided, defaults to GB.

Virtual Private Cloud networking

--network=NETWORK

The Compute Engine Network to which the environment will be connected. If a'Custom Subnet Network' is provided,--subnetwork must be specifiedas well.

This flag argument must be specified if any of the other arguments in this groupare specified.

--subnetwork=SUBNETWORK

The Compute Engine Subnetwork(https://cloud.google.com/compute/docs/subnetworks) to which the environmentwill be connected.

At most one of these can be specified:

--web-server-allow-all

Allows all IP addresses to access the Airflow web server.

--web-server-allow-ip=[description=DESCRIPTION],[ip_range=IP_RANGE]

Specifies a list of IPv4 or IPv6 ranges that will be allowed to access theAirflow web server. By default, all IPs are allowed to access the web server.

This is a repeated argument that can be specified multiple times to specifymultiple IP ranges. (e.g.--web-server-allow-ip=ip_range=130.211.160.0/28,description="officenetwork"--web-server-allow-ip=ip_range=130.211.114.0/28,description="legacynetwork")

ip_range

IPv4 or IPv6 range of addresses allowed to access the Airflow web server.

description

An optional description of the IP range.

--web-server-deny-all

Denies all incoming traffic to the Airflow web server.

Run$gcloud help for details.