Model monitoring overview

This document describes how BigQuery ML supports monitoring ofmachine learning (ML) models through evaluation and comparison the data a modeluses. This includes comparing a model's serving data to its training data,andcomparing new serving data to previously used serving data.

Understanding the data used by your models is a critical aspect of ML, becausethis data affects model performance. Understanding any variance betweenyour training and serving data is especially important in ensuring that yourmodels remain accurate over time. A model performs best on serving datathat is similar to the training data. When the serving data deviates from thedata used to train the model, the model's performance can deteriorate, even ifthe model itself hasn't changed.

BigQuery ML provides functions to help you analyze your trainingand serving data fordata skew anddata drift:

• Data skew occurs when the distribution of feature values for training datais significantly different from serving data in production. Trainingstatistics for the model are saved during model training, so the originaltraining data isn't required for you to use skew detection.
• Data drift occurs when feature data distribution in production changessignificantly over time. Drift detection is supported for consecutive spansof data, for example, between different days of serving data. This lets youget notified if the serving data is changing over time, before the data setsdiverge too much to retrain the model.

Use the following functions to monitor models in BigQuery ML:

• ML.DESCRIBE_DATA: computedescriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute thestatistics for a set of serving data, and then compare them to the trainingdata statistics that were computed when a BigQuery ML model wastrained, in order to identify anomalous differences between the two data sets.Statistics are only computed for feature columns in the serving data thatmatch feature columns in the training data, in order to achieve betterperformance and lower cost.
• ML.VALIDATE_DATA_DRIFT: compute andcompare the statistics for two sets of serving data in order toidentify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-graineddescriptive statistics for a set of training or serving data. This functionprovides the same behavior as theTensorFlowtfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compare thestatistics for training and serving data statistics, or two sets of serving data statistics,in order to identify anomalous differences between the two data sets.This function provides the same behavior as theTensorFlowvalidate_statistics API.

Monitoring use cases

This section describes how to use the BigQuery ML modelmonitoring functions in common monitoring use cases.

Basic data skew monitoring

This use case is appropriate when you want to quickly develop and monitor amodel for data skew and don't need fine-grained skew statistics tointegrate with an existing monitoring solution.

Typical steps for this use case are as follows:

1. Run theML.DESCRIBE_DATA function on your training and serving data,to make sure bothdata sets compare appropriately to each other and are within expectedparameters.
2. Create a BigQuery ML modeland train it on the training data.
3. Run theML.VALIDATE_DATA_SKEW function to compare the serving datastatistics with thetraining data statistics that were computed during model creation in orderto see if there's any data skew.
4. If there is data skew, investigate the root cause, adjust the trainingdata appropriately, and then retrain the model.

Basic data drift monitoring

This use case is appropriate when you want to quickly develop and monitor amodel for data drift and don't need fine-grained drift statistics tointegrate with an existing monitoring solution.

Typical steps for this use case are as follows:

1. Run theML.DESCRIBE_DATA function on your training and serving datato make sure bothdata sets compare appropriately to each other and are within expectedparameters.
2. Create a BigQuery ML modeland train it on the training data.
3. Run theML.VALIDATE_DATA_DRIFT function to compare the statisticsfor two differentserving data sets in order to see if there's any data drift. For example,you might want to compare the current serving data to historical serving datafrom atable snapshot, or to thefeatures served at a particular point in time, which you can get by using theML.FEATURES_AT_TIME function.
4. If there is data drift, investigate the root cause, adjust the trainingdata appropriately, and then retrain the model.

Advanced data skew or drift monitoring

This use case is appropriate when you want fine-grained skew or drift statisticsto integrate with an existing monitoring solution or for other purposes.

Typical steps for this use case are as follows:

1. Run theML.TFDV_DESCRIBE function on your training and serving dataat intervalsappropriate to your monitoring solution, andsave the query results. This step lets youcompare future serving data to training and serving data from past points intime.

2. Run theML.TFDV_VALIDATE function on your training and serving datastatistics, or on two sets of serving data statistics, to evaluate data skewor feature drift, respectively. The training and serving data must beprovided as a TensorFlowDatasetFeatureStatisticsList protocol bufferin JSON format. You can generate a protocol buffer in the correctformat by running theML.TFDV_DESCRIBE function, or you can load it fromoutside of BigQuery. The following example shows how to evaluatefeature skew:

   DECLAREstats1JSON;DECLAREstats2JSON;SETstats1=(SELECT*FROMML.TFDV_DESCRIBE(TABLE`myproject.mydataset.training`));SETstats2=(SELECT*FROMML.TFDV_DESCRIBE(TABLE`myproject.mydataset.serving`));SELECTML.TFDV_VALIDATE(stats1,stats2,'SKEW');INSERT`myproject.mydataset.serve_stats`(t,dataset_feature_statistics_list)SELECTCURRENT_TIMESTAMP()ASt,stats1;

3. If there is data skew or data drift, investigate the root cause,adjust the training data appropriately, and then retrain the model.

Monitoring visualization

Some monitoring functions offer integration withVertex AI model monitoring,so that you can use charts and graphs toanalyze model monitoring function output.

Using Vertex AI visualizations offers thefollowing benefits:

• Interactive visualizations: explore data distributions, skew metrics, anddrift metrics by using charts and graphs in the Vertex AIconsole.
• Historical analysis: track model monitoring results over time by usingVertex AI visualizations. This lets you identify trends andpatterns in data changes so that you can proactively update and maintainmodels.
• Centralized management: manage monitoring for all BigQuery MLand Vertex AI models in the unified Vertex AIdashboard.

You can enable visualization of theML.VALIDATE_DATA_DRIFT function outputby using that function'sMODEL argument. You can enable visualization of theML.VALIDATE_DATA_SKEW function output by using that function'senable_visualization_link argument.

You can only use monitoring visualization with models that areregistered withVertex AI. You can register an existing model by using theALTER MODEL statement.

Monitoring automation

You can automate monitoring by using ascheduled query to run themonitoring function, evaluate the output, and retrain the modelif anomalies are detected. You must enable email notifications as part ofsetting up the scheduled query.

For an example that shows how to automate theML.VALIDATE_DATA_SKEWfunction, seeAutomate skew detection.

What's next

For more information about supported SQL statements and functions for ML models,seeEnd-to-end user journey for ML models.