Vertex AI Pipelines lets you automate, monitor, and govern your machinelearning (ML) systems in a serverless manner by using ML pipelines toorchestrate your ML workflows. You can batch run ML pipelines defined usingthe Kubeflow Pipelines or the TensorFlow Extended(TFX) framework. To learn how to choose a framework fordefining your ML pipeline, seeInterfaces to define a pipeline.

This page provides an overview of the following:

• What is an ML pipeline?

• Structure of an ML pipeline

• Pipeline tasks and components

• Life cycle of an ML pipeline

• Use Vertex ML Metadata to track the lineage of ML artifacts

• Add pipeline runs to experiments

Note: If you're experienced in creating ML pipelines using the Kubeflow Pipelines SDK andwant to understand the differences between Vertex AI Pipelines andKubeflow Pipelines, seeMigrate from Kubeflow Pipelines toVertex AI Pipelines.

What is an ML pipeline?

An ML pipeline is a portable and extensible description of an MLOps workflow as aseries of steps called pipeline tasks. Each task performs a specificstep in the workflow to train and deploy an ML model.

With ML pipelines, you can apply MLOps strategies to automate and monitorrepeatable processes in your ML practice. For example, you can reuse a pipelinedefinition to continuously retrain a model on the latestproduction data. For more information about MLOps in Vertex AI,seeMLOps on Vertex AI.

Structure of an ML pipeline

An ML pipeline is a directed acyclic graph (DAG) of containerized pipeline tasksthat are interconnected using input-output dependencies. You canauthor each task either in Python or as a prebuilt container images.

You can define the pipeline as a DAG using either the Kubeflow PipelinesSDK or the TFX SDK, compile it to its YAML for intermediaterepresentation, and then run the pipeline. By default, pipeline tasks run inparallel. You can link the tasks to execute them in series.For more information about pipeline tasks, seePipeline task.For more information about the workflow for defining, compiling, and running thepipeline, seeLife cycle of an ML pipeline.

Click here for an example illustrating input-output dependencies between tasks within an ML pipeline.

Consider an ML pipeline with the following steps:

• Prepare data: Prepare or preprocess training data.

  • Input (from tasks within the same ML pipeline): None.

  • Output: Prepared or preprocessed training data.

• Train model: Use the prepared training data to train a model.

  • Input: Prepared or preprocessed training data from pipeline taskPrepare data.

  • Output: Trained model.

• Evaluate model: Evaluate the trained model.

  Input: Trained model from pipeline taskTrain model.

• Deploy: Deploy the trained model for predictions.

  Input: Trained model from pipeline taskTrain model.

When you compile your ML pipeline, the pipelines SDK you're using(Kubeflow Pipelines or TFX) analyzes the datadependencies between these tasks and creates the following workflow DAG:

• Prepare data doesn't rely on other tasks within the same ML pipeline forinputs. Therefore, it can be the first step in the ML pipeline, or runconcurrently with other tasks.

• Train model relies onPrepare data for inputs. Therefore, it occursafterPrepare data.

• Evaluate andDeploy both depend on the trained model. Therefore, theycan run concurrently, but afterTrain model.

When you run your ML pipeline, Vertex AI Pipelines executes these tasks in thesequence described in the DAG.

Pipeline tasks and components

Apipeline task is an instantiation of apipeline component with specific inputs. While definingyour ML pipeline, you can interconnect multiple tasks to form a DAG, by routingthe outputs of one pipeline task to the inputs for the next pipeline task inthe ML workflow. You can also use the inputs for the ML pipeline as the inputsfor a pipeline task.

Pipeline component

A pipeline component is a self-contained set of code that performs a specificstep of an ML workflow, such as data preprocessing, model training, ormodel deployment. A component typically consists of the following:

• Inputs: A component might have one or more input parameters and artifacts.

• Outputs: Every component has one or more output parameters or artifacts.

• Logic: This is the component's executable code. Forcontainerized components, the logic also contains the definition of theenvironment, or container image, where the component runs.

Components are the basis of defining tasks in an ML pipeline. To definepipeline tasks, you can either use predefinedGoogle Cloud Pipeline Components or create your own customcomponents.

Predefined components

Use predefined Google Cloud Pipeline Components if you want to use features of Vertex AI,such as AutoML, in your pipeline. To learn how to use Google Cloud Pipeline Componentsto define a pipeline, seeBuild a Pipeline.

Custom components

You can author your own custom components to use in your ML pipeline. For moreinformation about authoring custom components, seeBuild your own pipelinecomponents.

To learn how to author custom Kubeflow Pipelines components, see the"Pipelineswith lightweight components based on Python functions"tutorial notebook on GitHub. To learn how to author custom TFXcomponents, see theTFX Python function component tutorialon theTensorFlow Extended in Production tutorials.

Pipeline task

A pipeline task is the instantiation of a pipeline component and performs aspecific step in your ML workflow. You can author ML pipeline tasks either usingPython or as prebuilt container images.

Within a task, you can build on the on-demand compute capabilities of Vertex AIwith Kubernetes to scalably execute your code, or delegate your workload to anotherexecution engine, such as BigQuery, Dataflow, orGoogle Cloud Serverless for Apache Spark.

Lifecycle of an ML pipeline

From definition to execution and monitoring, the lifecycle of an ML pipelinecomprises the following high-level stages:

1. Define: The process of defining an ML pipeline and its task is alsocalled building a pipeline. In this stage, you need to perform the followingsteps:

   1. Choose an ML framework: Vertex AI Pipelines supports MLpipelines defined using the TFX or Kubeflow Pipelinesframework. To learn how to choose a framework for building your pipeline, seeInterfaces to define a pipeline.

   2. Define pipeline tasks and configure pipeline: For more information, seeBuild a Pipeline.

2. Compile: In this stage, you need to perform the following steps:

   1. Generate your ML pipeline definition in a compiled YAML file forintermediate representation, which you can use to run your ML pipeline.

   2. Optional: You can upload the compiled YAML file as apipeline templateto a repository and reuse it to create ML pipeline runs.

3. Run: Create an execution instance of your ML pipeline using the compiledYAML file or a pipeline template. The execution instance of a pipelinedefinition is called apipeline run.

   You can create a one-time occurrence of a pipeline run or use thescheduler APIto create recurring pipeline runs from the sameML pipeline definition. You can also clone an existing pipeline run. To learnhow to choose an interface to run an ML pipeline, seeInterfaces to run apipeline. For more information about how tocreate a pipeline run, seeRun a pipeline.

4. Monitor, visualize, and analyze runs: After you create a pipeline run,you can do the following to monitor the performance, status, and costs ofpipeline runs:

   • Configure email notifications for pipeline failures. For more information,seeConfigure email notifications.

   • Use Cloud Logging to create log entries for monitoring events. For moreinformation, seeView pipeline job logs.

   • Visualize, analyze, and compare pipeline runs. For more information, seeVisualize and analyze pipeline results.

   • Use Cloud Billing export to BigQuery to analyze pipeline runcosts. For more information, seeUnderstand pipeline run costs.

5. Optional: stop or delete pipeline runs: There is no restriction on howlong you can keep a pipeline run active. You can optionally do the following:

   • Stop a pipeline run.

   • Pause or resume a pipeline run schedule.

   • Delete an existing pipeline template, pipeline run, or pipeline run schedule.

What is a pipeline run?

A pipeline run is an execution instance of your ML pipeline definition. Eachpipeline run is identified by a unique run name. Using Vertex AI Pipelines,you can create an ML pipeline run in the following ways:

• Use the compiled YAML definition of a pipeline

• Use a pipeline template from the Template Gallery

For more information about how to create a pipeline run, seeRun apipeline. For more information about how tocreate a pipeline run from a pipeline template, seeCreate, upload, and use apipeline template.

For information about capturing and storing pipeline run metadatausing Vertex ML Metadata, seeUse Vertex ML Metadata to track thelineage of ML artifacts.

For information about using pipeline runs to experiment on your ML workflowusing Vertex AI Experiments, seeAdd your pipeline runs toexperiments.

Track the lineage of ML artifacts

A pipeline run contains several artifacts and parameters, including pipelinemetadata. To understand changes in the performance or accuracy of your MLsystem, you need to analyze the metadata and the lineage of ML artifacts fromyour ML pipeline runs. The lineage of an ML artifact includes all the factorsthat contributed to its creation, along with metadata and references to artifactsderived from it.

Lineage graphs help you analyze upstream root cause and downstream impact.Each pipeline run produces a lineage graph of parameters and artifacts that areinput into the run, materialized within the run, and output from the run.Metadata that composes this lineage graph is stored in Vertex ML Metadata.This metadata can also be synced to Dataplex Universal Catalog.

• Use Vertex ML Metadata to track pipeline artifact lineage

  When you run a pipeline using Vertex AI Pipelines, allparameters and artifact metadata consumed and generated by the pipeline are stored inVertex ML Metadata. Vertex ML Metadata is a managed implementation ofthe ML Metadata library in TensorFlow, and supports registering andwriting custom metadata schemas. When you create a pipeline run inVertex AI Pipelines, metadata from the pipeline run isstored in the default metadata store for the project and region whereyou execute the pipeline.

• Use Dataplex Universal Catalog to track pipeline artifact lineage

  Dataplex Universal Catalog is a global and cross-projectdata fabric integrated with multiple systems within Google Cloud, such asVertex AI, BigQuery, and Cloud Composer.Within Dataplex Universal Catalog, you can search for a pipeline artifact andview its lineage graph. Note that to prevent artifact conflicts, any resourcecatalogued in Dataplex Universal Catalog is identified with afully qualified name (FQN).

  Learn about Dataplex Universal Catalog usage costs.

For more information about tracking the lineage of ML artifacts usingVertex ML Metadata and Dataplex Universal Catalog, seeTrack the lineage of pipelineartifacts.

For more information about visualizing, analyzing, and comparing pipeline runs,seeVisualize and analyze pipeline results.For a list of first-party artifact types defined in Google Cloud Pipeline Components, seeML Metadata artifact types.

Add pipeline runs to experiments

Vertex AI Experiments lets you track and analyze various modelarchitectures, hyperparameters, and training environments to find the best modelfor your ML use case. After you create an ML pipeline run, you can associate itwith an experiment or experiment run. By doing so, you can experiment withdifferent sets of variables, such as hyperparameters, number of training steps,or iterations.

For more information about experimenting with ML workflows usingVertex AI Experiments, seeIntroduction toVertex AI Experiments.

What's next

• Learn about theinterfaces you can use to define and run pipelines using Vertex AI Pipelines.

• Get started bylearning how to define a pipeline using the Kubeflow Pipelines SDK.

• Learn how to run a pipeline.

• Learn aboutbest practices for implementing custom-trained ML models on Vertex AI.