gcloud dataflow flex-template build - builds a flex template file from the specified parameters

gcloud dataflow flex-template buildTEMPLATE_FILE_GCS_PATH--sdk-language=SDK_LANGUAGE(--image=IMAGE    |--env=[ENV,…]--flex-template-base-image=FLEX_TEMPLATE_BASE_IMAGE--image-gcr-path=IMAGE_GCR_PATH (--go-binary-path=GO_BINARY_PATH |--jar=[JAR,…] |--py-path=[PY_PATH,…])    | [--yaml-pipeline-path=YAML_PIPELINE_PATH :--yaml-image=YAML_IMAGE])[--additional-experiments=[ADDITIONAL_EXPERIMENTS,…]][--additional-user-labels=[ADDITIONAL_USER_LABELS,…]][--cloud-build-service-account=CLOUD_BUILD_SERVICE_ACCOUNT][--dataflow-kms-key=DATAFLOW_KMS_KEY][--disable-public-ips][--enable-streaming-engine][--gcs-log-dir=GCS_LOG_DIR][--image-repository-cert-path=IMAGE_REPOSITORY_CERT_PATH][--image-repository-password-secret-id=IMAGE_REPOSITORY_PASSWORD_SECRET_ID][--image-repository-username-secret-id=IMAGE_REPOSITORY_USERNAME_SECRET_ID][--max-workers=MAX_WORKERS][--metadata-file=PATH_TO_FILE][--network=NETWORK][--num-workers=NUM_WORKERS][--print-only][--service-account-email=SERVICE_ACCOUNT_EMAIL][--staging-location=STAGING_LOCATION][--subnetwork=SUBNETWORK][--temp-location=TEMP_LOCATION][--worker-machine-type=WORKER_MACHINE_TYPE][--worker-region=WORKER_REGION    |--worker-zone=WORKER_ZONE][GCLOUD_WIDE_FLAG …]

gclouddataflowflex-templatebuildgs://template-file-gcs-path--image=gcr://image-path--metadata-file=/local/path/to/metadata.json--sdk-language=JAVA

If using prebuilt template image from private registry, run:

gclouddataflowflex-templatebuildgs://template-file-gcs-path--image=private.registry.com:3000/image-path--image-repository-username-secret-id="projects/test-project/secrets/username-secret"--image-repository-password-secret-id="projects/test-project/secrets/password-secret/versions/latest"--metadata-file=metadata.json--sdk-language=JAVA

To build the template image and flex template JSON file, run:

gclouddataflowflex-templatebuildgs://template-file-gcs-path--image-gcr-path=gcr://path-to-store-image--jar=path/to/pipeline.jar--jar=path/to/dependency.jar--env=FLEX_TEMPLATE_JAVA_MAIN_CLASS=classpath--flex-template-base-image=JAVA11--metadata-file=/local/path/to/metadata.json--sdk-language=JAVA

TEMPLATE_FILE_GCS_PATH

The Google Cloud Storage location of the flex template file.Overrides if filealready exists.

--sdk-language=SDK_LANGUAGE

SDK language of the flex template job.SDK_LANGUAGE mustbe one of:JAVA,PYTHON,GO,YAML.

Exactly one of these must be specified:

--image=IMAGE

Path to the any image registry location of the prebuilt flex template image.

Or at least one of these can be specified:

--env=[ENV,…]

Environment variables to create for the Dockerfile. You can pass them as a commaseparated list or repeat individually with --env flag. Ex: --env="A=B,C=D" or--env A=B, --env C=D.When you reference files/dir in env variables, pleasespecify relative path to the paths passed via --py-path.Ex: if you pass.--py-path="path/pipleline/" then setFLEX_TEMPLATE_PYTHON_PY_FILE="pipeline/pipeline.py" You can find the list ofsupported environment variables in this link.https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#setting_required_dockerfile_environment_variables.

This flag argument must be specified if any of the other arguments in this groupare specified.

--flex-template-base-image=FLEX_TEMPLATE_BASE_IMAGE

Flex template base image to be used while building the container image. Allowedchoices are allowed labels (JAVA11/17/21/25, PYTHON3, GO), supported distrolessimages (JAVA11/17/21/25_DISTROLESS, GO_DISTROLESS), or full gcr.io path of thespecific version of the base image. For labels, we use the latest base imageversion to build the container. You can also provide a specific version fromthis linkhttps://gcr.io/dataflow-templates-base/

This flag argument must be specified if any of the other arguments in this groupare specified.

--image-gcr-path=IMAGE_GCR_PATH

The Google Container Registry or Google Artifact Registry location to store theflex template image to be built.

This flag argument must be specified if any of the other arguments in this groupare specified.

Exactly one of these must be specified:

--go-binary-path=GO_BINARY_PATH

Local path to your compiled dataflow pipeline Go binary. The binary should becompiled to run on the target worker architecture (usually linux-amd64). Seehttps://beam.apache.org/documentation/sdks/go-cross-compilation/for more information.

--jar=[JAR,…]

Local path to your dataflow pipeline jar file and all their dependent jar filesrequired for the flex template classpath. You can pass them as a comma separatedlist or repeat individually with --jar flag. Ex: --jar="code.jar,dep.jar" or--jar code.jar, --jar dep.jar.

--py-path=[PY_PATH,…]

Local path to your dataflow pipeline python files and all their dependent filesrequired for the flex template classpath. You can pass them as a comma separatedlist or repeat individually with --py-path flag. Ex:--py-path="path/pipleline/,path/dependency/" or --py-path path/pipleline/,--py-path path/dependency/.

Or at least one of these can be specified:

--yaml-pipeline-path=YAML_PIPELINE_PATH

Local path to your YAML pipeline file. Use a full or relative path to a localfile containing the value of yaml_pipeline_path.

This flag argument must be specified if any of the other arguments in this groupare specified.

--yaml-image=YAML_IMAGE

Path to the any image registry location of the prebuilt yaml template image.

--additional-experiments=[ADDITIONAL_EXPERIMENTS,…]

Default experiments to pass to the job.

--additional-user-labels=[ADDITIONAL_USER_LABELS,…]

Default user labels to pass to the job. Example:--additional-user-labels='{"key1":"value1"}'

--cloud-build-service-account=CLOUD_BUILD_SERVICE_ACCOUNT

Service account to run the Cloud Build in the formatprojects/{project}/serviceAccounts/{service_account}. Ensure that the accountyou are using to run 'gcloud dataflow flex-template build' has'ServiceAccountUser' role on the specified Cloud Build service account youprovide with the --cloud-build-service-account flag. The specified serviceaccount must have required permissions to build the image. If the specifiedservice account is in a project that is different from the project where you arestarting builds, seehttps://cloud.google.com/build/docs/securing-builds/configure-user-specified-service-accounts#cross-project_set_upto grant the necessary access.

--dataflow-kms-key=DATAFLOW_KMS_KEY

Default Cloud KMS key to protect the job resources.

--disable-public-ips

Cloud Dataflow workers must not use public IP addresses. Overrides the defaultdataflow/disable_public_ips property value for this commandinvocation.

--enable-streaming-engine

Enable Streaming Engine for the streaming job by default. Overrides the defaultdataflow/enable_streaming_engine property value for this commandinvocation.

--gcs-log-dir=GCS_LOG_DIR

Google Cloud Storage directory to save build logs.(Must be a URL beginning with'gs://'.)

--image-repository-cert-path=IMAGE_REPOSITORY_CERT_PATH

The full URL to self-signed certificate of private registry in Cloud Storage.For example, gs://mybucket/mycerts/selfsigned.crt. The certificate provided inCloud Storage must be DER-encoded and may be supplied in binary or printable(Base64) encoding. If the certificate is provided in Base64 encoding, it must bebounded at the beginning by -----BEGIN CERTIFICATE-----, and must be bounded atthe end by -----END CERTIFICATE-----. If this parameter is provided, the dockerdaemon in the template launcher will be instructed to trust that certificate.

--image-repository-password-secret-id=IMAGE_REPOSITORY_PASSWORD_SECRET_ID

Secret Manager secret id for the password to authenticate to private registry.Should be in the formatprojects/{project}/secrets/{secret}/versions/{secret_version} orprojects/{project}/secrets/{secret}. If the version is not provided latestversion will be used.

--image-repository-username-secret-id=IMAGE_REPOSITORY_USERNAME_SECRET_ID

Secret Manager secret id for the username to authenticate to private registry.Should be in the formatprojects/{project}/secrets/{secret}/versions/{secret_version} orprojects/{project}/secrets/{secret}. If the version is not provided latestversion will be used.

--max-workers=MAX_WORKERS

Default maximum number of workers to run.

--metadata-file=PATH_TO_FILE

Local path to the metadata json file for the flex template. Use a full orrelative path to a local file containing the value of metadata_file.

--network=NETWORK

Default Compute Engine network for launching instances to run your pipeline.

--num-workers=NUM_WORKERS

Initial number of workers to use by default.

--print-only

Prints the container spec to stdout. Does not save in Google Cloud Storage.Overrides the defaultdataflow/print_only property value for thiscommand invocation.

--service-account-email=SERVICE_ACCOUNT_EMAIL

Default service account to run the workers as.

--staging-location=STAGING_LOCATION

Default Google Cloud Storage location to stage local files.(Must be a URLbeginning with 'gs://'.)

--subnetwork=SUBNETWORK

Default Compute Engine subnetwork for launching instances to run your pipeline.

--temp-location=TEMP_LOCATION

Default Google Cloud Storage location to stage temporary files. If not set,defaults to the value for --staging-location.(Must be a URL beginning with'gs://'.)

--worker-machine-type=WORKER_MACHINE_TYPE

Default type of machine to use for workers. Defaults to server-specified.

At most one of these can be specified:

--worker-region=WORKER_REGION

Default region to run the workers in.

--worker-zone=WORKER_ZONE

Default zone to run the workers in.

Run$gcloud help for details.