gcloud ai custom-jobs create - create a new custom job

gcloud ai custom-jobs create--display-name=DISPLAY_NAME(--config=CONFIG--worker-pool-spec=[WORKER_POOL_SPEC,…])[--args=[ARG,…]][--command=[COMMAND,…]][--enable-dashboard-access][--enable-web-access][--labels=[KEY=VALUE,…]][--network=NETWORK][--persistent-resource-id=PERSISTENT_RESOURCE_ID][--python-package-uris=[PYTHON_PACKAGE_URIS,…]][--region=REGION][--service-account=SERVICE_ACCOUNT][--kms-key=KMS_KEY :--kms-keyring=KMS_KEYRING--kms-location=KMS_LOCATION--kms-project=KMS_PROJECT][GCLOUD_WIDE_FLAG …]

--display-name=DISPLAY_NAME

Display name of the custom job to create.

Worker pool specification.

At least one of these must be specified:

--config=CONFIG

Path to the job configuration file. This file should be a YAML documentcontaining a`CustomJobSpec`.If an option is specified both in the configuration file **and** viacommand-line arguments, the command-line arguments override the configurationfile. Note that keys with underscore are invalid.

Example(YAML):

workerPoolSpecs:machineSpec:machineType:n1-highmem-2replicaCount:1containerSpec:imageUri:gcr.io/ucaip-test/ucaip-training-testargs:-port=8500command:-start

--worker-pool-spec=[WORKER_POOL_SPEC,…]

Define the worker pool configuration used by the custom job. You can specifymultiple worker pool specs in order to create a custom job with multiple workerpools.

The spec can contain the following fields:

machine-type

(Required): The type of the machine. seehttps://cloud.google.com/vertex-ai/docs/training/configure-compute#machine-typesfor supported types. This is corresponding to themachineSpec.machineType field inWorkerPoolSpec APImessage.

replica-count

The number of worker replicas to use for this worker pool, by default the valueis 1. This is corresponding to thereplicaCount field inWorkerPoolSpec API message.

accelerator-type

The type of GPUs. seehttps://cloud.google.com/vertex-ai/docs/training/configure-compute#specifying_gpusfor more requirements. This is corresponding to themachineSpec.acceleratorType field inWorkerPoolSpecAPI message.

accelerator-count

The number of GPUs for each VM in the worker pool to use, by default the valueif 1. This is corresponding to themachineSpec.acceleratorCountfield inWorkerPoolSpec API message.

container-image-uri

The URI of a container image to be directly run on each worker replica. This iscorresponding to thecontainerSpec.imageUri field inWorkerPoolSpec API message.

executor-image-uri

The URI of a container image that will run the provided package.

output-image-uri

The URI of a custom container image to be built for autopackaged custom jobs.

python-module

The Python module name to run within the provided package.

local-package-path

The local path of a folder that contains training code.

script

The relative path under thelocal-package-path to a file toexecute. It can be a Python file or an arbitrary bash script.

requirements

Python dependencies to be installed from PyPI, separated by ";". This issupposed to be used when some public packages are required by your trainingapplication but not in the base images. It has the same effect as editing a"requirements.txt" file underlocal-package-path.

extra-packages

Relative paths of local Python archives to be installed, separated by ";". Thisis supposed to be used when some custom packages are required by your trainingapplication but not in the base images. Every path should be relative to thelocal-package-path.

extra-dirs

Relative paths of the folders underlocal-package-path to be copiedinto the container, separated by ";". If not specified, only the parentdirectory that contains the main executable (script orpython-module) will be copied.

Note that some of these fields are used for different job creation methods andare categorized as mutually exclusive groups listed below. Exactly one of thesegroups of fields must be specified:

container-image-uri

Specify this field to use a custom container image for training. Together withthe--command and--args flags, this field representsa`WorkerPoolSpec.ContainerSpec`message. In this case, the--python-package-uris flag isdisallowed.

Example:--worker-pool-spec=replica-count=1,machine-type=n1-highmem-2,container-image-uri=gcr.io/ucaip-test/ucaip-training-test

executor-image-uri, python-module

Specify these fields to train using a pre-built container and Python packagesthat are already in Cloud Storage. Together with the--python-package-uris and--args flags, these fieldsrepresent a`WorkerPoolSpec.PythonPackageSpec`message .

Example:--worker-pool-spec=machine-type=e2-standard-4,executor-image-uri=us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-4:latest,python-module=trainer.task

output-image-uri

Specify this field to push the output custom container training image to aspecific path in Container Registry or Artifact Registry for an autopackagedcustom job.

Example:--worker-pool-spec=machine-type=e2-standard-4,executor-image-uri=us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-4:latest,output-image-uri='eu.gcr.io/projectName/imageName',python-module=trainer.task

local-package-path, executor-image-uri, output-image-uri, python-module|script

Specify these fields, optionally withrequirements,extra-packages, orextra-dirs, to train using apre-built container and Python code from a local path. In this case, the--python-package-uris flag is disallowed.

Example usingpython-module:--worker-pool-spec=machine-type=e2-standard-4,replica-count=1,executor-image-uri=us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-4:latest,python-module=trainer.task,local-package-path=/usr/page/application

Example usingscript:--worker-pool-spec=machine-type=e2-standard-4,replica-count=1,executor-image-uri=us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-4:latest,script=my_run.sh,local-package-path=/usr/jeff/application

--args=[ARG,…]

Comma-separated arguments passed to containers or python tasks.

--command=[COMMAND,…]

Command to be invoked when containers are started. It overrides the entrypointinstruction in Dockerfile when provided.

--enable-dashboard-access

Whether you want Vertex AI to enable dashboard built on the training containers.If set totrue, you can access thedashboard at the URIs given by CustomJob.web_access_uris orTrial.web_access_uris (within HyperparameterTuningJob.trials).

--enable-web-access

Whether you want Vertex AI to enableinteractiveshell access to training containers. If set totrue, you can access interactive shells atthe URIs given by CustomJob.web_access_uris or Trial.web_access_uris (withinHyperparameterTuningJob.trials).

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

--network=NETWORK

Full name of the Google Compute Engine network to which the Job is peered with.Private services access must already have been configured. If unspecified, theJob is not peered with any network.

--persistent-resource-id=PERSISTENT_RESOURCE_ID

The name of the persistent resource from the same project and region on which torun this custom job.

If this is specified, the job will be run on existing machines held by thePersistentResource instead of on-demand short-lived machines. The network andCMEK configs on the job should be consistent with those on thePersistentResource, otherwise, the job will be rejected.

--python-package-uris=[PYTHON_PACKAGE_URIS,…]

The common Python package URIs to be used for training with a pre-builtcontainer image. e.g.--python-package-uri=path1,path2 If you areusing multiple worker pools and want to specify a different Python packag foreach pool, use--config instead.

Region resource - Cloud region to create a custom job. This represents a Cloudresource. (NOTE) Some attributes are not given arguments in this group but canbe set in other ways.

To set theproject attribute:

• provide the argument--region on the command line with a fullyspecified name;
• set the propertyai/region with a fully specified name;
• choose one from the prompted list of available regions with a fully specifiedname;
• provide the argument--project on the command line;
• set the propertycore/project.

--region=REGION

ID of the region or fully qualified identifier for the region.

To set theregion attribute:

• provide the argument--region on the command line;
• set the propertyai/region;
• choose one from the prompted list of available regions.

--service-account=SERVICE_ACCOUNT

The email address of a service account to use when running the trainingappplication. You must have theiam.serviceAccounts.actAspermission for the specified service account.

Key resource - The Cloud KMS (Key Management Service) cryptokey that will beused to protect the custom job. The 'Vertex AI Service Agent' service accountmust hold permission 'Cloud KMS CryptoKey Encrypter/Decrypter'. The arguments inthis group can be used to specify the attributes of this resource.

--kms-key=KMS_KEY

ID of the key or fully qualified identifier for the key.

To set thekms-key attribute:

• provide the argument--kms-key on the command line.

This flag argument must be specified if any of the other arguments in this groupare specified.

--kms-keyring=KMS_KEYRING

The KMS keyring of the key.

To set thekms-keyring attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-keyring on the command line.

--kms-location=KMS_LOCATION

The Google Cloud location for the key.

To set thekms-location attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-location on the command line.

--kms-project=KMS_PROJECT

The Google Cloud project for the key.

To set thekms-project attribute:

• provide the argument--kms-key on the command line with a fullyspecified name;
• provide the argument--kms-project on the command line;
• set the propertycore/project.

Run$gcloud help for details.