Before you use Vertex AI Pipelines to orchestrate your machine learning(ML) pipelines, you must set up your Google Cloud project. Some resources, such asthe metadata store used by Vertex ML Metadata, are created in yourGoogle Cloud project the first time thatyou run a pipeline.

Use the following instructions to configure your project forVertex AI Pipelines.

1. Create your Google Cloud project and configure it for use withVertex AI Pipelines.

2. If you don't specify a service account, Vertex AI Pipelinesuses the Compute Engine default service account to run your pipelines.For more information about the Compute Engine default service account, seeUsing the Compute Engine Default Service Account.

   We recommend that youcreate a service account to run your pipelines andthen grant this account granular permissions to the Google Cloud resourcesthat are needed to run your pipeline.

3. Vertex AI Pipelines uses Cloud Storage to store theartifacts of your pipeline runs.Create a Cloud Storage bucketand grant your service account access to this bucket.

4. Vertex AI Pipelines uses Vertex ML Metadata to store themetadata created by your pipeline runs. When you run a pipeline for the first time, if the metadata store of your project doesn't exist,Vertex AI creates your project's metadata store.

   If you want your data encrypted using a customer-managed encryption key(CMEK), you can manuallycreate your metadata store using aCMEK key before you run a pipeline. Otherwise,if there's no existing default metadata store in your project,Vertex AI creates your project's metadata store using theCMEK key used when you run the pipeline for the first time. After themetadata store is created, it uses a CMEK key that'sdifferent from the CMEK key used in a pipeline run.

Set up your Google Cloud project

Use the following instructions to create a Google Cloud project and configureit for use with Vertex AI Pipelines.

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Vertex AI, Compute Engine, and Cloud Storage APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

After initializing the gcloud CLI, update it and install the required components:

gcloudcomponentsupdategcloudcomponentsinstallbeta

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Vertex AI, Compute Engine, and Cloud Storage APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

After initializing the gcloud CLI, update it and install the required components:

gcloudcomponentsupdategcloudcomponentsinstallbeta

Configure a service account with granular permissions

When you run a pipeline, you can specify a service account. Your pipeline runacts with the permissions of this service account.

If you don't specify a service account, your pipeline run uses theCompute Enginedefault service account. For more information about the Compute Engine default service account, seeUsing the Compute Engine Default Service Account.

• Use the following instructions to create a service account and grant itgranular permissions to Google Cloud resources.

  1. Run the following command to create a service account.

     gcloud iam service-accounts createSERVICE_ACCOUNT_ID \    --description="DESCRIPTION" \    --display-name="DISPLAY_NAME" \    --project=PROJECT_ID

     Replace the following values:

     • SERVICE_ACCOUNT_ID: The ID for the service account.
     • DESCRIPTION: (Optional.) A description of the serviceaccount.
     • DISPLAY_NAME: The display name for this serviceaccount.
     • PROJECT_ID: The project to create your service accountin.

     Learn more aboutcreating a service account.

  2. Grant your service account access to Vertex AI.Note that it might take some time for the access change to propagate.For more information, seeAccess change propagation.

     gcloud projects add-iam-policy-bindingPROJECT_ID \    --member="serviceAccount:SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com" \    --role="roles/aiplatform.user"

     Replace the following values:

     • PROJECT_ID: The project that your service account wascreated in.
     • SERVICE_ACCOUNT_ID: The ID for the service account.
     Note: The minimum set of permissions that are required to useVertex AI Pipelines to run a pipeline areaiplatform.metadataStores.get,storage.buckets.get,storage.objects.create, andstorage.objects.get. The first time you run a pipeline in a newproject you must also add theaiplatform.metadataStores.createpermission. To run a pipeline with minimalpermissions, use Identity and Access Management tocreate a custom rolewiththese permissions, and grant your service account access to that roleinstead ofroles/aiplatform.user.

  3. You can use Artifact Registry to host container images and Kubeflow Pipelinestemplates.

     For more information about Artifact Registry, see theArtifact Registry documentation.

  4. Grant your service account access to any Google Cloud resources that you use in your pipelines.

     gcloud projects add-iam-policy-bindingPROJECT_ID \    --member="serviceAccount:SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com" \    --role="ROLE_NAME"

     Replace the following values:

     • PROJECT_ID: The project that your service account wascreated in.
     • SERVICE_ACCOUNT_ID: The ID for the service account.
     • ROLE_NAME: The Identity and Access Management role to grant to thisservice account.

  5. To use Vertex AI Pipelines to run pipelines with thisservice account, run the following command to grant your user accounttheroles/iam.serviceAccountUser role for your service account.

     gcloud iam service-accounts add-iam-policy-binding \SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com \    --member="user:USER_EMAIL" \    --role="roles/iam.serviceAccountUser"

     Replace the following values:

     • SERVICE_ACCOUNT_ID: The ID for the service account.
     • PROJECT_ID: The project that your service account wascreated in.
     • USER_EMAIL: The email address of the user that runspipelines as this service account.

• If you prefer to use the Compute Engine default service accountto run your pipelines, enable the Compute Engine API and grant your default service account access to Vertex AI.Note that it might take some time for the access change to propagate.For more information, seeAccess change propagation.

  gcloud projects add-iam-policy-bindingPROJECT_ID \    --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \    --role="roles/aiplatform.user"

  Replace the following values:

  • PROJECT_ID: The project that your default service account wascreated in.
  • PROJECT_NUMBER: The project number that your default serviceaccount was created in.

  Roles required to enable APIs

  To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

  Enable the API

Configure a Cloud Storage bucket for pipeline artifacts

Vertex AI Pipelines stores the artifacts of your pipeline runsusing Cloud Storage. Use the following instructions to create aCloud Storage bucket and grant your service account (or the Compute Engine default service account) access to read andwrite objects in that bucket.

1. Run the following command to create a Cloud Storage bucket in theregion that you want to run your pipelines in.

   gcloud storage buckets create gs://BUCKET_NAME --location=BUCKET_LOCATION --project=PROJECT_ID

   Replace the following values:

   • PROJECT_ID: Specify the project that your bucket isassociated with.
   • BUCKET_LOCATION: Specify thelocation ofyour bucket — for example,US-CENTRAL1.
   • BUCKET_NAME: The name you want to give your bucket, subjecttonaming requirements. For example,my-bucket.

   Learn more aboutcreating Cloud Storagebuckets.

2. Run the following commands to grant your service account access to readand write pipeline artifacts in the bucket that you created in the previousstep.

   gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \--member=serviceAccount:SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com \--role=roles/storage.objectUser

   Replace the following values:

   • SERVICE_ACCOUNT_ID: The ID for the service account.
   • PROJECT_ID: The project that your service account wascreated in.
   • BUCKET_NAME: The name of the bucket you are granting yourservice account access to.

   Alternatively, if you prefer to use the Compute Engine default service accountto run your pipelines, run thegcloud iam service-accounts list command to locate the project number for that account.

   gcloud iam service-accounts list

   The Compute Engine default service account is named like the following:PROJECT_NUMBER-compute@developer.gserviceaccount.com.

   Run the following command to grant the Compute Engine default service account access to readand write pipeline artifacts in the bucket that you created in the previousstep.

   gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \--member=serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \--role=roles/storage.objectUser

   Replace the following values:

   • PROJECT_NUMBER: The project number for the Compute Engine default service account.
   • BUCKET_NAME: The name of the bucket you are granting yourservice account access to.

   Learn more aboutcontrolling access to Cloud Storagebuckets.

Create a metadata store that uses a CMEK (optional)

Use the following instructions to create a CMEK and set up aVertex ML Metadata metadata store that uses this CMEK.

1. Use Cloud Key Management Service to configure a customer-managed encryptionkey.

2. Use the following REST call to create your project's default metadata storeusing your CMEK.

   Before using any of the request data, make the following replacements:

   • LOCATION_ID: Your region.
   • PROJECT_ID: Yourproject ID.
   • KEY_RING: The name of the Cloud Key Management Service key ring that your encryption key is on.
   • KEY_NAME: The name of the encryption key that you want to use for this metadata store.

   HTTP method and URL:

   POST https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/metadataStores?metadata_store_id=default

   Request JSON body:

   {  "encryption_spec": {    "kms_key_name": "projects/PROJECT_ID/locations/LOCATION_ID/keyRings/KEY_RING/cryptoKeys/KEY_NAME"  },}

   To send your request, expand one of these options:

   curl (Linux, macOS, or Cloud Shell)

   Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login , or by usingCloud Shell, which automatically logs you into thegcloud CLI . You can check the currently active account by runninggcloud auth list.

   Save the request body in a file namedrequest.json. Run the following command in the terminal to create or overwrite this file in the current directory:

   cat > request.json<< 'EOF'{  "encryption_spec": {    "kms_key_name": "projects/PROJECT_ID/locations/LOCATION_ID/keyRings/KEY_RING/cryptoKeys/KEY_NAME"  },}EOF

   Then execute the following command to send your REST request:

   curl -X POST \
        -H "Authorization: Bearer $(gcloud auth print-access-token)" \
        -H "Content-Type: application/json; charset=utf-8" \
        -d @request.json \
        "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/metadataStores?metadata_store_id=default"

   PowerShell (Windows)

   Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login . You can check the currently active account by runninggcloud auth list.

   Save the request body in a file namedrequest.json. Run the following command in the terminal to create or overwrite this file in the current directory:

   @'{  "encryption_spec": {    "kms_key_name": "projects/PROJECT_ID/locations/LOCATION_ID/keyRings/KEY_RING/cryptoKeys/KEY_NAME"  },}'@  | Out-File -FilePath request.json -Encoding utf8

   Then execute the following command to send your REST request:

   $cred = gcloud auth print-access-token
   $headers = @{ "Authorization" = "Bearer $cred" }
   
   Invoke-WebRequest `
       -Method POST `
       -Headers $headers `
       -ContentType: "application/json; charset=utf-8" `
       -InFile request.json `
       -Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/metadataStores?metadata_store_id=default" | Select-Object -Expand Content

   You should receive a JSON response similar to the following:

   {  "name": "projects/PROJECT_ID/locations/LOCATION_ID/operations/OPERATIONS_ID",  "metadata": {    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.CreateMetadataStoreOperationMetadata",    "genericMetadata": {      "createTime": "2021-05-18T18:47:14.494997Z",      "updateTime": "2021-05-18T18:47:14.494997Z"    }  }}