Model inference overview

This document describes the types of batch inference that BigQuery MLsupports, which include:

• Batch prediction
• Online prediction

Machine learning inference is the process of running data points intoa machine learning model to calculate an output such as a single numericalscore. This process is also referred to as "operationalizing a machine learningmodel" or "putting a machine learning model into production."

Batch prediction

The following sections describe the available ways of performing prediction inBigQuery ML.

Inference using BigQuery ML trained models

Prediction in BigQuery ML is used not only for supervised learning models, butalso unsupervised learning models.

BigQuery ML supports prediction functionalities through theML.PREDICT function,with the following models:

Inference using imported models

With this approach, you create and train a model outside ofBigQuery, import it by using theCREATE MODEL statement,and then run inference on it by using theML.PREDICT function.All inference processing occurs in BigQuery, using data fromBigQuery. Imported models can perform supervised orunsupervised learning.

BigQuery ML supports the followingtypes of imported models:

• Open Neural Network Exchange (ONNX) for models trained in PyTorch, scikit-learn, and other popular ML frameworks.
• TensorFlow
• TensorFlow Lite
• XGBoost

Use this approach to make use of custom models developed with a rangeof ML frameworks while taking advantage of BigQuery ML'sinference speed and co-location with data.

To learn more, try one of the following tutorials:

• Make predictions with imported TensorFlow models
• Make predictions with scikit-learn models in ONNX format
• Make predictions with PyTorch models in ONNX format

Inference using remote models

With this approach, you can create a reference to a modelhosted inVertex AI Inferenceby using theCREATE MODEL statement,and then run inference on it by using theML.PREDICT function.All inference processing occurs in Vertex AI, using data fromBigQuery. Remote models can perform supervised orunsupervised learning.

Use this approach to run inference against large models that require the GPUhardware support provided by Vertex AI. If most of yourmodels are hosted by Vertex AI, this also lets you runinference against these models by using SQL, without having to manually builddata pipelines to take data to Vertex AI and bring predictionresults back to BigQuery.

For step-by-step instructions, seeMake predictions with remote models on Vertex AI.

Batch inference with BigQuery models in Vertex AI

BigQuery ML has built-in support for batch prediction, without theneed to use Vertex AI. It is also possible to register aBigQuery ML model to Model Registry in order toperform batch prediction in Vertex AI using aBigQuery table as input. However, this can onlybe done by using the Vertex AI API and settingInstanceConfig.instanceTypetoobject.

Online prediction

The built-in inference capability of BigQuery ML is optimized forlarge-scale use cases, such as batch prediction. While BigQuery MLdelivers low latency inference results when handling small input data, you canachieve faster online prediction through seamless integration withVertex AI.

You canmanage BigQuery ML models within the Vertex AI environment,which eliminates the need to export models from BigQuery ML beforedeploying them as Vertex AI endpoints. By managing models withinVertex AI, you get access to all of the Vertex AI MLOpscapabilities, and also to features such asVertex AI Feature Store.

Additionally, you have the flexibility toexport BigQuery ML models toCloud Storage for availability on other model hosting platforms.

What's next

• For more information about using Vertex AI models togenerate text and embeddings, seeGenerative AI overview.
• For more information about using Cloud AI APIs to perform AI tasks, seeAI application overview.

• For more information about supported SQL statements and functions fordifferent model types, see the following documents:

  • End-to-end user journeys for generative AI models
  • End-to-end user journeys for time series forecasting models
  • End-to-end user journeys for ML models
  • End-to-end user journeys for imported models