Console

1. In the Google Cloud console, in the Vertex AI section, goto theTraining pipelines page.

   Go to Training pipelines

2. Clickadd_boxCreate to open theTrain new model pane.

3. On theTraining method step, specify the following settings:

   1. If you want touse a managed dataset fortraining, then specify aDataset and anAnnotation set.

      Otherwise, in theDataset drop-down list, selectNo manageddataset.

   2. SelectCustom training (advanced).

   ClickContinue.

4. On theModel details step, chooseTrain new model orTrain new version.If you select train new model, enter a name of your choice,MODEL_NAME, for your model. ClickContinue.

5. On theTraining container step, specify the following settings:

   1. Selectwhether to use aPrebuilt container or aCustomcontainerfor training.

   2. Depending on your choice, do one of the following:

      • If you want to use a prebuilt container for training, then provideVertex AI with information it needs to use the trainingpackage that you have uploaded to Cloud Storage:

        1. Use theModel framework andModel framework versiondrop-down lists to specify theprebuiltcontainer that you want touse.

        2. In thePackage location field, specify theCloud Storage URI of thePython training application thatyou have created anduploaded. Thisfile usually ends with.tar.gz.

        3. In thePython module field, enter themodule name of yourtraining application's entrypoint.

      • If you want to use acustom container fortraining, then in theContainer image field, specify the Artifact Registryor Docker Hub URI of your container image.

   3. In theModel output directory field,specify the Cloud Storage URI of a directory in abucket that you have access to. The directory does not need to exist yet.

      This value gets passed to Vertex AI in thebaseOutputDirectory APIfield, which setsseveral environment variables that your training application can accesswhen it runs.

      At the end of training, Vertex AI looks formodelartifacts in a subdirectory ofthis URI in order to create aModel. (This subdirectory is available toyour training code as theAIP_MODEL_DIR environment variable.)

      When you don't use hyperparametertuning, Vertex AI expects to find model artifacts inBASE_OUTPUT_DIRECTORY/model/.

   4. Optional: In theArguments field, you can specify arguments forVertex AI to use when it starts running your training code.The maximum length for all arguments combined is 100,000 characters.The behavior of these arguments differs depending on what type ofcontainer you are using:

      • If you are using a prebuilt container, then Vertex AIpasses the arguments as command-line flags to yourPython module.

      • If you are using a custom container, then Vertex AIoverrides your container'sCMD instruction with thearguments.

   ClickContinue.

6. On theHyperparameter tuning step, make sure that theEnablehyperparameter tuning checkbox is not selected. ClickContinue.

7. On theCompute and pricing step, specify the following settings:

   1. In theRegion drop-down list, select a "region that supports customtraining"

   2. In theWorker pool 0 section, specifycomputeresources to use for training.

      If you specify accelerators,make sure the type of accelerator that youchoose is available in your selectedregion.

      If you want to performdistributedtraining, then clickAdd moreworker pools and specify an additional set of compute resources foreach additional worker pool that you want.

   ClickContinue.

8. On thePrediction container step, specify the followingsettings:

   1. Selectwhether to use aPrebuilt container or aCustomcontainerto serve predictions from your trained model.

   2. Depending on your choice, do one of the following:

      • If you want to use a prebuilt container to serve predictions, thenuse theModel framework,Model framework version, andAccelerator type fields to choosewhich prebuilt predictioncontainer to use for prediction.

        MatchModel framework andModel framework version to themachine learning framework you used for training. Only specify anAccelerator type if you want to lateruse GPUs for online orbatch predictions.

      • If you want to use a custom container to serve predictions, then dothe following:

        1. In theContainer image field, specify theArtifact Registry URI of your containerimage.

        2. Optionally, you may specify aCommand tooverride thecontainer'sENTRYPOINTinstruction.

   3. TheModel directory field contains the value that you previously setin theModel output directory field of theTraining containerstep. Changing either of these fields has the same effect. See theprevious instruction for moreinformation about this field.

   4. Leave the fields in thePredict schemata section blank.

9. ClickStart training to start the serverless training pipeline.

REST

Use the following code sample to create a training pipeline using thecreate method of thetrainingPipeline resource.

Note: If you want to set this pipeline to create a new modelversion,you can optionally add thePARENT_MODELin thetrainingPipeline field.

To learn more, seeModel versioning with Vertex AI Model Registry.

Before using any of the request data, make the following replacements:

• LOCATION_ID: The region where the training code is run and theModel is stored.
• PROJECT_ID: Your project ID.
• TRAINING_PIPELINE_NAME: Required. A display name for the trainingPipeline.
• If your training application uses a Vertex AI dataset, specify the following:
  • DATASET_ID: The ID of the dataset.
  • ANNOTATIONS_FILTER: Filters the dataset by the annotations that you specify.
  • ANNOTATION_SCHEMA_URI: Filters the dataset by the specified annotation schema URI.
  • Use one of the following options to specify how data items are split into training, validation, and test sets.
    • To split the dataset based on fractions defining the size of each set, specify the following:
      • TRAINING_FRACTION: The fraction of the dataset to use to train your model.
      • VALIDATION_FRACTION: The fraction of the dataset to use to validate your model.
      • TEST_FRACTION: The fraction of the dataset to use to evaluate your model.
    • To split the dataset based on filters, specify the following:
      • TRAINING_FILTER: Filters the dataset to data items to use for training your model.
      • VALIDATION_FILTER: Filters the dataset to data items to use for validating your model.
      • TEST_FILTER: Filters the dataset to data items to use for evaluating your model.
    • To use a predefined split, specify the following:
      • PREDEFINED_SPLIT_KEY: The name of the column to use to split the dataset. Acceptable values in this column include `training`, `validation`, and `test`.
    • To split the dataset based on the timestamp on the dataitems, specify the following:
      • TIMESTAMP_TRAINING_FRACTION: The fraction of the dataset to use to train your model.
      • TIMESTAMP_VALIDATION_FRACTION: The fraction of the dataset to use to validate your model.
      • TIMESTAMP_TEST_FRACTION: The fraction of the dataset to use to evaluate your model.
      • TIMESTAMP_SPLIT_KEY: The name of the timestamp column to use to split the dataset.
  • OUTPUT_URI_PREFIX: The Cloud Storage location where Vertex AI exports your training dataset, once it has been split into training, validation, and test sets.
• Define the custom training job:
  • MACHINE_TYPE: The type of the machine. Refer toavailable machine types for training.
  • ACCELERATOR_TYPE: (Optional.) The type of accelerator to attach to each trial.
  • ACCELERATOR_COUNT: (Optional.) The number of accelerators to attach to each trial.
  • REPLICA_COUNT: The number of worker replicas to use for each trial.
  • If your training application runs in a custom container, specify the following:
    • CUSTOM_CONTAINER_IMAGE_URI: The URI of a container image in Artifact Registry or Docker Hub that is to be run on each worker replica.
    • CUSTOM_CONTAINER_COMMAND: (Optional.) The command to be invoked when the container is started. This command overrides the container's default entrypoint.
    • CUSTOM_CONTAINER_ARGS: (Optional.) The arguments to be passed when starting the container. The maximum length for all arguments combined is 100,000 characters.
  • If your training application is a Python package that runs in a prebuilt container, specify the following:
    • PYTHON_PACKAGE_EXECUTOR_IMAGE_URI: The URI of the container image that runs the provided Python package. Refer to theavailable prebuilt containers for training.
    • PYTHON_PACKAGE_URIS: The Cloud Storage location of the Python package files which are the training program and its dependent packages. The maximum number of package URIs is 100.
    • PYTHON_MODULE: The Python module name to run after installing the packages.
    • PYTHON_PACKAGE_ARGS: (Optional.) Command-line arguments to be passed to the Python module. The maximum length for all arguments combined is 100,000 characters.
  • TIMEOUT: (Optional.) The maximum running time for the job.
• MODEL_NAME: A display name for the model uploaded (created) by the TrainingPipeline.
• MODEL_DESCRIPTION: A description for the model.
• IMAGE_URI: The URI of the container image to use for running predictions. For example,us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-1:latest. Useprebuilt containers orcustom containers.
• modelToUpload.labels: Any set of key-value pairs to organize your models. For example:
  • "env": "prod"
  • "tier": "backend"
• Specify theLABEL_NAME andLABEL_VALUE for any labels that you want to apply to this training pipeline.

HTTP method and URL:

POST https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines

Request JSON body:

{  "displayName": "TRAINING_PIPELINE_NAME",  "inputDataConfig": {    "datasetId":DATASET_ID,    "annotationsFilter":ANNOTATIONS_FILTER,    "annotationSchemaUri":ANNOTATION_SCHEMA_URI,    // Union field split can be only one of the following:    "fractionSplit": {      "trainingFraction":TRAINING_FRACTION,      "validationFraction":VALIDATION_FRACTION,      "testFraction":TEST_FRACTION    },    "filterSplit": {      "trainingFilter":TRAINING_FILTER,      "validationFilter":VALIDATION_FILTER,      "testFilter":TEST_FILTER    },    "predefinedSplit": {      "key":PREDEFINED_SPLIT_KEY    },    "timestampSplit": {      "trainingFraction":TIMESTAMP_TRAINING_FRACTION,      "validationFraction":TIMESTAMP_VALIDATION_FRACTION,      "testFraction":TIMESTAMP_TEST_FRACTION,      "key":TIMESTAMP_SPLIT_KEY    }    // End of list of possible types for union field split.    "gcsDestination": {      "outputUriPrefix":OUTPUT_URI_PREFIX    }  },  "trainingTaskDefinition": "gs://google-cloud-aiplatform/schema/trainingjob/definition/custom_task_1.0.0.yaml",  "trainingTaskInputs": {    "workerPoolSpecs": [        {          "machineSpec": {            "machineType":MACHINE_TYPE,            "acceleratorType":ACCELERATOR_TYPE,            "acceleratorCount":ACCELERATOR_COUNT          },          "replicaCount":REPLICA_COUNT,          // Union field task can be only one of the following:          "containerSpec": {            "imageUri":CUSTOM_CONTAINER_IMAGE_URI,            "command": [CUSTOM_CONTAINER_COMMAND            ],            "args": [CUSTOM_CONTAINER_ARGS            ]          },          "pythonPackageSpec": {            "executorImageUri":PYTHON_PACKAGE_EXECUTOR_IMAGE_URI,            "packageUris": [PYTHON_PACKAGE_URIS            ],            "pythonModule":PYTHON_MODULE,            "args": [PYTHON_PACKAGE_ARGS            ]          }          // End of list of possible types for union field task.        }      ],      "scheduling": {        "TIMEOUT":TIMEOUT      }    }  },  "modelToUpload": {    "displayName": "MODEL_NAME",    "predictSchemata": {},    "containerSpec": {      "imageUri": "IMAGE_URI"    }  },  "labels": {LABEL_NAME_1":LABEL_VALUE_1,LABEL_NAME_2":LABEL_VALUE_2  }}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines" | Select-Object -Expand Content

The response contains information about specifications as well as theTRAININGPIPELINE_ID.

{  "name": "projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines/TRAININGPIPELINE_ID",  "displayName": "TRAINING_PIPELINE_NAME",  "input_data_config" {    "dataset_id": "1234567891011121314"    "gcs_destination" {      "output_uri_prefix": "gs://BUCKET_NAME/data/20200915191342"    }    "annotation_schema_uri": "gs://google-cloud-aiplatform/schema/dataset/annotation/image_classification_1.0.0.yaml"  },  "trainingTaskDefinition": "gs://google-cloud-aiplatform/schema/trainingjob/definition/custom_task_1.0.0.yaml",  "trainingTaskInputs": {    "workerPoolSpecs": [      {        "machineSpec": {          "machineType": "n1-standard-4"        },        "replicaCount": "1",        "pythonPackageSpec": {          "executorImageUri": "us-docker.pkg.dev/vertex-ai/training/training-tf-cpu.2-1:latest",          "packageUris": [            "gs://BUCKET_NAME/training/hello-custom-training-1.0.tar.gz"          ],          "pythonModule": "trainer.task",          "args": [            "--model-dir=gs://BUCKET_NAME/output/"          ]        }      }    ]  },  "trainingTaskMetadata": {    "backingCustomJob": "projects/PROJECT_ID/locations/LOCATION_ID/customJobs/CUSTOM_JOB_ID"  },  "modelToUpload": {    "displayName": "MODEL_NAME",    "predictSchemata": {},    "containerSpec": {      "imageUri": "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-1:latest"    }  },  "state": "PIPELINE_STATE_PENDING",  "createTime": "2020-09-15T19:09:54.342080Z",  "startTime": "2020-09-15T19:13:42.991045Z",}

Java

Before trying this sample, follow theJava setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AIJava API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

importcom.google.cloud.aiplatform.v1.LocationName;importcom.google.cloud.aiplatform.v1.Model;importcom.google.cloud.aiplatform.v1.ModelContainerSpec;importcom.google.cloud.aiplatform.v1.PipelineServiceClient;importcom.google.cloud.aiplatform.v1.PipelineServiceSettings;importcom.google.cloud.aiplatform.v1.TrainingPipeline;importcom.google.gson.JsonArray;importcom.google.gson.JsonObject;importcom.google.protobuf.Value;importcom.google.protobuf.util.JsonFormat;importjava.io.IOException;publicclassCreateTrainingPipelineCustomJobSample{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.Stringproject="PROJECT";StringdisplayName="DISPLAY_NAME";StringmodelDisplayName="MODEL_DISPLAY_NAME";StringcontainerImageUri="CONTAINER_IMAGE_URI";StringbaseOutputDirectoryPrefix="BASE_OUTPUT_DIRECTORY_PREFIX";createTrainingPipelineCustomJobSample(project,displayName,modelDisplayName,containerImageUri,baseOutputDirectoryPrefix);}staticvoidcreateTrainingPipelineCustomJobSample(Stringproject,StringdisplayName,StringmodelDisplayName,StringcontainerImageUri,StringbaseOutputDirectoryPrefix)throwsIOException{PipelineServiceSettingssettings=PipelineServiceSettings.newBuilder().setEndpoint("us-central1-aiplatform.googleapis.com:443").build();Stringlocation="us-central1";// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests. After completing all of your requests, call// the "close" method on the client to safely clean up any remaining background resources.try(PipelineServiceClientclient=PipelineServiceClient.create(settings)){JsonObjectjsonMachineSpec=newJsonObject();jsonMachineSpec.addProperty("machineType","n1-standard-4");// A working docker image can be found at// gs://cloud-samples-data/ai-platform/mnist_tfrecord/custom_job// This sample image accepts a set of arguments including model_dir.JsonObjectjsonContainerSpec=newJsonObject();jsonContainerSpec.addProperty("imageUri",containerImageUri);JsonArrayjsonArgs=newJsonArray();jsonArgs.add("--model_dir=$(AIP_MODEL_DIR)");jsonContainerSpec.add("args",jsonArgs);JsonObjectjsonJsonWorkerPoolSpec0=newJsonObject();jsonJsonWorkerPoolSpec0.addProperty("replicaCount",1);jsonJsonWorkerPoolSpec0.add("machineSpec",jsonMachineSpec);jsonJsonWorkerPoolSpec0.add("containerSpec",jsonContainerSpec);JsonArrayjsonWorkerPoolSpecs=newJsonArray();jsonWorkerPoolSpecs.add(jsonJsonWorkerPoolSpec0);JsonObjectjsonBaseOutputDirectory=newJsonObject();// The GCS location for outputs must be accessible by the project's AI Platform// service account.jsonBaseOutputDirectory.addProperty("output_uri_prefix",baseOutputDirectoryPrefix);JsonObjectjsonTrainingTaskInputs=newJsonObject();jsonTrainingTaskInputs.add("workerPoolSpecs",jsonWorkerPoolSpecs);jsonTrainingTaskInputs.add("baseOutputDirectory",jsonBaseOutputDirectory);Value.BuildertrainingTaskInputsBuilder=Value.newBuilder();JsonFormat.parser().merge(jsonTrainingTaskInputs.toString(),trainingTaskInputsBuilder);ValuetrainingTaskInputs=trainingTaskInputsBuilder.build();StringtrainingTaskDefinition="gs://google-cloud-aiplatform/schema/trainingjob/definition/custom_task_1.0.0.yaml";StringimageUri="gcr.io/cloud-aiplatform/prediction/tf-cpu.1-15:latest";ModelContainerSpeccontainerSpec=ModelContainerSpec.newBuilder().setImageUri(imageUri).build();ModelmodelToUpload=Model.newBuilder().setDisplayName(modelDisplayName).setContainerSpec(containerSpec).build();TrainingPipelinetrainingPipeline=TrainingPipeline.newBuilder().setDisplayName(displayName).setTrainingTaskDefinition(trainingTaskDefinition).setTrainingTaskInputs(trainingTaskInputs).setModelToUpload(modelToUpload).build();LocationNameparent=LocationName.of(project,location);TrainingPipelineresponse=client.createTrainingPipeline(parent,trainingPipeline);System.out.format("response: %s\n",response);System.out.format("Name: %s\n",response.getName());}}}

Python

To learn how to install or update the Vertex AI SDK for Python, seeInstall the Vertex AI SDK for Python. For more information, see thePython API reference documentation.

The following examples show how to use theVertex AI SDK for Python to create aserverless trainingpipeline. Choose whether you plan to use acustomcontainer or aprebuiltcontainer for training:

defcreate_training_pipeline_custom_package_job_sample(project:str,location:str,staging_bucket:str,display_name:str,python_package_gcs_uri:str,python_module_name:str,container_uri:str,model_serving_container_image_uri:str,dataset_id:Optional[str]=None,model_display_name:Optional[str]=None,args:Optional[List[Union[str,float,int]]]=None,replica_count:int=1,machine_type:str="n1-standard-4",accelerator_type:str="ACCELERATOR_TYPE_UNSPECIFIED",accelerator_count:int=0,training_fraction_split:float=0.8,validation_fraction_split:float=0.1,test_fraction_split:float=0.1,sync:bool=True,tensorboard_resource_name:Optional[str]=None,service_account:Optional[str]=None,):aiplatform.init(project=project,location=location,staging_bucket=staging_bucket)job=aiplatform.CustomPythonPackageTrainingJob(display_name=display_name,python_package_gcs_uri=python_package_gcs_uri,python_module_name=python_module_name,container_uri=container_uri,model_serving_container_image_uri=model_serving_container_image_uri,)# This example uses an ImageDataset, but you can use another typedataset=aiplatform.ImageDataset(dataset_id)ifdataset_idelseNonemodel=job.run(dataset=dataset,model_display_name=model_display_name,args=args,replica_count=replica_count,machine_type=machine_type,accelerator_type=accelerator_type,accelerator_count=accelerator_count,training_fraction_split=training_fraction_split,validation_fraction_split=validation_fraction_split,test_fraction_split=test_fraction_split,sync=sync,tensorboard=tensorboard_resource_name,service_account=service_account,)model.wait()print(model.display_name)print(model.resource_name)print(model.uri)returnmodel

defcreate_training_pipeline_custom_job_sample(project:str,location:str,staging_bucket:str,display_name:str,script_path:str,container_uri:str,model_serving_container_image_uri:str,dataset_id:Optional[str]=None,model_display_name:Optional[str]=None,args:Optional[List[Union[str,float,int]]]=None,replica_count:int=0,machine_type:str="n1-standard-4",accelerator_type:str="ACCELERATOR_TYPE_UNSPECIFIED",accelerator_count:int=0,training_fraction_split:float=0.8,validation_fraction_split:float=0.1,test_fraction_split:float=0.1,sync:bool=True,tensorboard_resource_name:Optional[str]=None,service_account:Optional[str]=None,):aiplatform.init(project=project,location=location,staging_bucket=staging_bucket)job=aiplatform.CustomTrainingJob(display_name=display_name,script_path=script_path,container_uri=container_uri,model_serving_container_image_uri=model_serving_container_image_uri,)# This example uses an ImageDataset, but you can use another typedataset=aiplatform.ImageDataset(dataset_id)ifdataset_idelseNonemodel=job.run(dataset=dataset,model_display_name=model_display_name,args=args,replica_count=replica_count,machine_type=machine_type,accelerator_type=accelerator_type,accelerator_count=accelerator_count,training_fraction_split=training_fraction_split,validation_fraction_split=validation_fraction_split,test_fraction_split=test_fraction_split,sync=sync,tensorboard=tensorboard_resource_name,service_account=service_account,)model.wait()print(model.display_name)print(model.resource_name)print(model.uri)returnmodel

defcreate_training_pipeline_custom_container_job_sample(project:str,location:str,staging_bucket:str,display_name:str,container_uri:str,model_serving_container_image_uri:str,dataset_id:Optional[str]=None,model_display_name:Optional[str]=None,args:Optional[List[Union[str,float,int]]]=None,replica_count:int=1,machine_type:str="n1-standard-4",accelerator_type:str="ACCELERATOR_TYPE_UNSPECIFIED",accelerator_count:int=0,training_fraction_split:float=0.8,validation_fraction_split:float=0.1,test_fraction_split:float=0.1,sync:bool=True,tensorboard_resource_name:Optional[str]=None,service_account:Optional[str]=None,):aiplatform.init(project=project,location=location,staging_bucket=staging_bucket)job=aiplatform.CustomContainerTrainingJob(display_name=display_name,container_uri=container_uri,model_serving_container_image_uri=model_serving_container_image_uri,)# This example uses an ImageDataset, but you can use another typedataset=aiplatform.ImageDataset(dataset_id)ifdataset_idelseNonemodel=job.run(dataset=dataset,model_display_name=model_display_name,args=args,replica_count=replica_count,machine_type=machine_type,accelerator_type=accelerator_type,accelerator_count=accelerator_count,training_fraction_split=training_fraction_split,validation_fraction_split=validation_fraction_split,test_fraction_split=test_fraction_split,sync=sync,tensorboard=tensorboard_resource_name,service_account=service_account,)model.wait()print(model.display_name)print(model.resource_name)print(model.uri)returnmodel

REST

Use the following code sample to create a training pipeline using thecreate method of thetrainingPipeline resource.

Before using any of the request data, make the following replacements:

• LOCATION_ID: Your project's region.
• PROJECT_ID: Your project ID.
• TRAINING_PIPELINE_NAME: Required. A display name for the trainingPipeline.
• If your training application uses a Vertex AI dataset, specify the following:
  • DATASET_ID: The ID of the dataset.
  • ANNOTATIONS_FILTER: Filters the dataset by the annotations that you specify.
  • ANNOTATION_SCHEMA_URI: Filters the dataset by the specified annotation schema URI.
  • Use one of the following options to specify how data items are split into training, validation, and test sets.
    • To split the dataset based on fractions defining the size of each set, specify the following:
      • TRAINING_FRACTION: The fraction of the dataset to use to train your model.
      • VALIDATION_FRACTION: The fraction of the dataset to use to validate your model.
      • TEST_FRACTION: The fraction of the dataset to use to evaluate your model.
    • To split the dataset based on filters, specify the following:
      • TRAINING_FILTER: Filters the dataset to data items to use for training your model.
      • VALIDATION_FILTER: Filters the dataset to data items to use for validating your model.
      • TEST_FILTER: Filters the dataset to data items to use for evaluating your model.
    • To use a predefined split, specify the following:
      • PREDEFINED_SPLIT_KEY: The name of the column to use to split the dataset. Acceptable values in this column include `training`, `validation`, and `test`.
    • To split the dataset based on the timestamp on the dataitems, specify the following:
      • TIMESTAMP_TRAINING_FRACTION: The fraction of the dataset to use to train your model.
      • TIMESTAMP_VALIDATION_FRACTION: The fraction of the dataset to use to validate your model.
      • TIMESTAMP_TEST_FRACTION: The fraction of the dataset to use to evaluate your model.
      • TIMESTAMP_SPLIT_KEY: The name of the timestamp column to use to split the dataset.
  • OUTPUT_URI_PREFIX: The Cloud Storage location where Vertex AI exports your training dataset, after it has been split into training, validation, and test sets.
• Specify your hyperparameter tuning job:
  • Specify your metrics:
    • METRIC_ID: The name of this metric.
    • METRIC_GOAL: The goal of this metric. Can beMAXIMIZE orMINIMIZE.
  • Specify your hyperparameters:
    • PARAMETER_ID: The name of this hyperparameter.
    • PARAMETER_SCALE: (Optional.) How the parameter should be scaled. Leave unset for CATEGORICAL parameters. Can beUNIT_LINEAR_SCALE,UNIT_LOG_SCALE,UNIT_REVERSE_LOG_SCALE, orSCALE_TYPE_UNSPECIFIED
    • If this hyperparameter's type is DOUBLE, specify the minimum (DOUBLE_MIN_VALUE) and maximum (DOUBLE_MAX_VALUE) values for this hyperparameter.
    • If this hyperparameter's type is INTEGER, specify the minimum (INTEGER_MIN_VALUE) and maximum (INTEGER_MAX_VALUE) values for this hyperparameter.
    • If this hyperparameter's type is CATEGORICAL, specify the acceptable values (CATEGORICAL_VALUES) as an array of strings.
    • If this hyperparameter's type is DISCRETE, specify the acceptable values (DISCRETE_VALUES) as an array of numbers.
  • ALGORITHM: (Optional.) The search algorithm to use in this hyperparameter tuning job. Can beALGORITHM_UNSPECIFIED,GRID_SEARCH, orRANDOM_SEARCH.
  • MAX_TRIAL_COUNT: The maximum number of trials to run in this job.
  • PARALLEL_TRIAL_COUNT: The maximum number of trials that can run in parallel.
  • MAX_FAILED_TRIAL_COUNT: The number of jobs that can fail before the hyperparameter tuning job fails.
  • Define the trial custom training job:
    • MACHINE_TYPE: The type of the machine. Refer to theavailable machine types for training.
    • ACCELERATOR_TYPE: (Optional.) The type of accelerator to attach to each trial.
    • ACCELERATOR_COUNT: (Optional.) The number of accelerators to attach to each trial.
    • REPLICA_COUNT: The number of worker replicas to use for each trial.
    • If your training application runs in a custom container, specify the following:
      • CUSTOM_CONTAINER_IMAGE_URI: The URI of a container image in Artifact Registry or Docker Hub that is to be run on each worker replica.
      • CUSTOM_CONTAINER_COMMAND: (Optional.) The command to be invoked when the container is started. This command overrides the container's default entrypoint.
      • CUSTOM_CONTAINER_ARGS: (Optional.) The arguments to be passed when starting the container.
    • If your training application is a Python package that runs in a prebuilt container, specify the following:
      • PYTHON_PACKAGE_EXECUTOR_IMAGE_URI: The URI of the container image that runs the provided Python package. Refer to theavailable prebuilt containers for training.
      • PYTHON_PACKAGE_URIS: The Cloud Storage location of the Python package files which are the training program and its dependent packages. The maximum number of package URIs is 100.
      • PYTHON_MODULE: The Python module name to run after installing the packages.
      • PYTHON_PACKAGE_ARGS: (Optional.) Command-line arguments to be passed to the Python module.
  • Learn aboutjob scheduling options.
  • TIMEOUT: (Optional.) The maximum running time for each trial.
  • Specify theLABEL_NAME andLABEL_VALUE for any labels that you want to apply to this hyperparameter tuning job.
• MODEL_NAME: A display name for the model uploaded (created) by the TrainingPipeline.
• MODEL_DESCRIPTION: Optional. A description for the model.
• PREDICTION_IMAGE_URI: Required. Specify one of the two following options:
  • The image URI of theprebuilt container to use for prediction, such as "tf2-cpu.2-1:latest".
  • The image URI of your owncustom container to use for prediction.
• modelToUpload.labels: Optional. Any set of key-value pairs to organize your models. For example:
  • "env": "prod"
  • "tier": "backend"
• Specify theLABEL_NAME andLABEL_VALUE for any labels that you want to apply to this training pipeline.

HTTP method and URL:

POST https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines

Request JSON body:

{  "displayName": "TRAINING_PIPELINE_NAME",  "inputDataConfig": {    "datasetId":DATASET_ID,    "annotationsFilter":ANNOTATIONS_FILTER,    "annotationSchemaUri":ANNOTATION_SCHEMA_URI,    // Union field split can be only one of the following:    "fractionSplit": {      "trainingFraction":TRAINING_FRACTION,      "validationFraction":VALIDATION_FRACTION,      "testFraction":TEST_FRACTION    },    "filterSplit": {      "trainingFilter":TRAINING_FILTER,      "validationFilter":VALIDATION_FILTER,      "testFilter":TEST_FILTER    },    "predefinedSplit": {      "key":PREDEFINED_SPLIT_KEY    },    "timestampSplit": {      "trainingFraction":TIMESTAMP_TRAINING_FRACTION,      "validationFraction":TIMESTAMP_VALIDATION_FRACTION,      "testFraction":TIMESTAMP_TEST_FRACTION,      "key":TIMESTAMP_SPLIT_KEY    }    // End of list of possible types for union field split.    "gcsDestination": {      "outputUriPrefix":OUTPUT_URI_PREFIX    }  },  "trainingTaskDefinition": "gs://google-cloud-aiplatform/schema/trainingjob/definition/hyperparameter_tuning_task_1.0.0.yaml",  "trainingTaskInputs": {    "studySpec": {    "metrics": [      {        "metricId":METRIC_ID,        "goal":METRIC_GOAL      }    ],    "parameters": [      {        "parameterId":PARAMETER_ID,        "scaleType":PARAMETER_SCALE,        // Union field parameter_value_spec can be only one of the following:        "doubleValueSpec": {            "minValue":DOUBLE_MIN_VALUE,            "maxValue":DOUBLE_MAX_VALUE        },        "integerValueSpec": {            "minValue":INTEGER_MIN_VALUE,            "maxValue":INTEGER_MAX_VALUE        },        "categoricalValueSpec": {            "values": [CATEGORICAL_VALUES            ]        },        "discreteValueSpec": {            "values": [DISCRETE_VALUES            ]        }        // End of list of possible types for union field parameter_value_spec.      }    ],    "ALGORITHM":ALGORITHM  },  "maxTrialCount":MAX_TRIAL_COUNT,  "parallelTrialCount":PARALLEL_TRIAL_COUNT,  "maxFailedTrialCount":MAX_FAILED_TRIAL_COUNT,  "trialJobSpec": {      "workerPoolSpecs": [        {          "machineSpec": {            "machineType":MACHINE_TYPE,            "acceleratorType":ACCELERATOR_TYPE,            "acceleratorCount":ACCELERATOR_COUNT          },          "replicaCount":REPLICA_COUNT,          // Union field task can be only one of the following:          "containerSpec": {            "imageUri":CUSTOM_CONTAINER_IMAGE_URI,            "command": [CUSTOM_CONTAINER_COMMAND            ],            "args": [CUSTOM_CONTAINER_ARGS            ]          },          "pythonPackageSpec": {            "executorImageUri":PYTHON_PACKAGE_EXECUTOR_IMAGE_URI,            "packageUris": [PYTHON_PACKAGE_URIS            ],            "pythonModule":PYTHON_MODULE,            "args": [PYTHON_PACKAGE_ARGS            ]          }          // End of list of possible types for union field task.        }      ],      "scheduling": {        "TIMEOUT":TIMEOUT      }    },    "labels": {LABEL_NAME_1":LABEL_VALUE_1,LABEL_NAME_2":LABEL_VALUE_2    }  },  "modelToUpload": {    "displayName": "MODEL_NAME",    "description": "MODEL_DESCRIPTION",    "predictSchemata": {},    "containerSpec": {      "imageUri": "PREDICTION_IMAGE_URI"    }  },  "labels": {LABEL_NAME_1":LABEL_VALUE_1,LABEL_NAME_2":LABEL_VALUE_2  }}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines" | Select-Object -Expand Content

The response contains information about specifications as well as theTRAININGPIPELINE_ID.

{  "name": "projects/PROJECT_ID/locations/LOCATION_ID/trainingPipelines/TRAININGPIPELINE_ID",  "displayName": "TRAINING_PIPELINE_NAME",  "input_data_config" {    "dataset_id": "1234567891011121314"    "gcs_destination" {      "output_uri_prefix": "gs://BUCKET_NAME/data/20200915191342"    }    "annotation_schema_uri": "gs://google-cloud-aiplatform/schema/dataset/annotation/image_classification_1.0.0.yaml"  },  "trainingTaskDefinition": "gs://google-cloud-aiplatform/schema/trainingjob/definition/custom_task_1.0.0.yaml",  "trainingTaskInputs": {    "name": "projects/12345/locations/us-central1/hyperparameterTuningJobs/6789",  "displayName": "myHyperparameterTuningJob",  "studySpec": {    "metrics": [      {        "metricId": "myMetric",        "goal": "MINIMIZE"      }    ],    "parameters": [      {        "parameterId": "myParameter1",        "integerValueSpec": {          "minValue": "1",          "maxValue": "128"        },        "scaleType": "UNIT_LINEAR_SCALE"      },      {        "parameterId": "myParameter2",        "doubleValueSpec": {          "minValue": 1e-07,          "maxValue": 1        },        "scaleType": "UNIT_LINEAR_SCALE"      }    ],    "ALGORITHM": "RANDOM_SEARCH"  },  "maxTrialCount": 20,  "parallelTrialCount": 1,  "trialJobSpec": {    "workerPoolSpecs": [      {        "machineSpec": {          "machineType": "n1-standard-4"        },        "replicaCount": "1",        "pythonPackageSpec": {          "executorImageUri": "us-docker.pkg.dev/vertex-ai/training/training-tf-cpu.2-1:latest",          "packageUris": [            "gs://my-bucket/my-training-application/trainer.tar.bz2"          ],          "pythonModule": "my-trainer.trainer"        }      }    ]  },  "state": "PIPELINE_STATE_PENDING",  "createTime": "2020-09-15T19:09:54.342080Z",  "startTime": "2020-09-15T19:13:42.991045Z",}