Vertex AI Experiments is a tool that helps you track and analyze differentmodel architectures, hyperparameters, and training environments,letting you track the steps, inputs, and outputs ofan experiment run. Vertex AI Experiments can also evaluate how your model performed in aggregate,against test datasets, and during the training run. You can then use thisinformation to select the best model for your particular use case.

Experiment runs don't incur additional charges. You're only charged forresources that you use during your experiment as described inVertex AI pricing.

What do you want to do?	Check out notebook sample
track metrics and parameters	Compare models
track experiment lineage	Model training
track pipeline runs	Compare pipeline runs

Track steps, inputs, and outputs

Vertex AI Experiments lets you track:

• steps of anexperiment run,for example, preprocessing, training,
• inputs, for example, algorithm, parameters, datasets,
• outputs of those steps, for example, models, checkpoints, metrics.

You can then figure out what worked and what didn't, and identify furtheravenues for experimentation.

For user journey examples, check out:

• Model training
• Compare models

Analyze model performance

Vertex AI Experiments lets you track and evaluate howthe model performed in aggregate, against test datasets, and duringthe training run. This ability helps to understand the performancecharacteristics of the models -- how well a particular model works overall,where it fails, and where the model excels.

For user journey examples, check out:

• Compare pipeline runs
• Compare models

Compare model performance

Vertex AI Experiments lets you group and compare multiple modelsacrossexperiment runs.Each model has its own specified parameters, modeling techniques, architectures,and input. This approach helps select the best model.

For user journey examples, check out:

• Compare pipeline runs
• Compare models

Search experiments

The Google Cloud console provides a centralized view of experiments,a cross-sectional view of the experiment runs, and the details for each run.The Vertex AI SDK for Python provides APIs to consume experiments, experiment runs,experiment run parameters, metrics, and artifacts.

Vertex AI Experiments, along withVertex ML Metadata, provides a wayto find the artifacts tracked in an experiment. This lets you quickly view theartifact's lineage and the artifacts consumed and produced by steps in a run.

Scope of support

Vertex AI Experiments supports development of models usingVertex AI custom training, Vertex AI Workbenchnotebooks, Notebooks, and all Python ML Frameworks across most ML Frameworks.For some ML frameworks, such as TensorFlow, Vertex AI Experimentsprovides deep integrations into the framework that makes the user experienceautomagical. For other ML frameworks, Vertex AI Experiments providesa framework neutral Vertex AI SDK for Python that you can use.(see:Prebuilt containers forTensorFlow, scikit-learn, PyTorch, XGBoost).

Data models and concepts

Vertex AI Experiments is acontextinVertex ML Metadata where an experimentcan containn experiment runs in addition ton pipeline runs. An experimentrun consists of parameters, summary metrics, time series metrics, andPipelineJob,Artifact, andExecution Vertex AI resources.Vertex AI TensorBoard, amanaged version of open source TensorBoard, is used for time-series metricsstorage. Executions andartifacts of a pipeline run are viewablein theGoogle Cloud console.

Vertex AI Experiments terms

Experiment, experiment run, and pipeline run

experiment

• An experiment is a context that can contain a set of n experiment runs in addition to pipeline runs where a user can investigate, as a group, different configurations such as input artifacts or hyperparameters.

SeeCreate an experiment.

experiment run

• A specific, trackable execution within a Vertex AI Experiment, which logs inputs (like algorithm, parameters, and datasets) and outputs (like models, checkpoints, and metrics) to monitor and compare ML development iterations. For more information, seeCreate and manage experiment runs.

SeeCreate and manage experiment runs.

pipeline run

• One or more Vertex PipelineJobs can be associated with an experiment where each PipelineJob is represented as a single run. In this context, the parameters of the run are inferred by the parameters of the PipelineJob. The metrics are inferred from the system.Metric artifacts produced by that PipelineJob. The artifacts of the run are inferred from artifacts produced by that PipelineJob.

One or more Vertex AIPipelineJob resourcecan be associated with anExperimentRun resource.In this context, the parameters, metrics, and artifacts are not inferred.

SeeAssociate a pipeline with an experiment.

Parameters and metrics

SeeLog parameters.

summary metrics

• Summary metrics are a single value for each metric key in an experiment run. For example, the test accuracy of an experiment is the accuracy calculated against a test dataset at the end of training that can be captured as a single value summary metric.

SeeLog summary metrics.

time series metrics

• Time series metrics are longitudinal metric values where each value represents a step in the training routine portion of a run. Time series metrics are stored in Vertex AI TensorBoard. Vertex AI Experiments stores a reference to the Vertex TensorBoard resource.

SeeLog time series metrics.

Resource types

pipeline job

• A pipeline job or a pipeline run corresponds to the PipelineJob resource in the Vertex AI API. It's an execution instance of your ML pipeline definition, which is defined as a set of ML tasks interconnected by input-output dependencies.

artifact

• An artifact is a discrete entity or piece of data produced and consumed by a machine learning workflow. Examples of artifacts include datasets, models, input files, and training logs.

Vertex AI Experiments lets you use a schema to define the type ofartifact. For example, supported schema types includesystem.Dataset,system.Model, andsystem.Artifact. For more information, seeSystem schemas.

Notebook tutorial

• Get started with Vertex AI Experiments

What's next

• Set up to get started with Vertex AI Experiments