The Keras library provides a way to calculate and report on a suite of standard metrics when training deep learning models.
In addition to offering standard metrics for classification and regression problems, Keras also allows you to define and report on your own custom metrics when training deep learning models. This is particularly useful if you want to keep track of a performance measure that better captures the skill of your model during training.
In this tutorial, you will discover how to use the built-in metrics and how to define and use your own metrics when training deep learning models in Keras.
After completing this tutorial, you will know:
How Keras metrics work and how you can use them when training your models.
How to use regression and classification metrics in Keras with worked examples.
How to define and use your own custom metric in Keras with a worked example.
Kick-start your project with my new bookDeep Learning With Python, includingstep-by-step tutorials and thePython source code files for all examples.
Let’s get started.
Update Jan/2020: Updated API for Keras 2.3 and TensorFlow 2.0.
Metrics and How to Use Custom Metrics for Deep Learning with Keras in Python Photo byIndi Samarajiva, some rights reserved.
Tutorial Overview
This tutorial is divided into 4 parts; they are:
Keras Metrics
Keras Regression Metrics
Keras Classification Metrics
Custom Metrics in Keras
Keras Metrics
Keras allows you to list the metrics to monitor during the training of your model.
You can do this by specifying the “metrics” argument and providing a list of function names (or function name aliases) to thecompile() function on your model.
For example:
1
model.compile(...,metrics=['mse'])
The specific metrics that you list can be the names of Keras functions (likemean_squared_error) or string aliases for those functions (like ‘mse‘).
Metric values are recorded at the end of eachepoch on the training dataset. If a validation dataset is also provided, then the metric recorded is also calculated for the validation dataset.
All metrics are reported in verbose output and in the history object returned from calling thefit() function. In both cases, the name of the metric function is used as the key for the metric values. In the case of metrics for the validation dataset, the “val_” prefix is added to the key.
Bothloss functions and explicitly defined Keras metrics can be used as training metrics.
Keras Regression Metrics
Below is a list of the metrics that you can use in Keras on regression problems.
Mean Squared Error: mean_squared_error, MSE or mse
Mean Absolute Error: mean_absolute_error, MAE, mae
Mean Absolute Percentage Error: mean_absolute_percentage_error, MAPE, mape
Cosine Proximity: cosine_proximity, cosine
The example below demonstrates these 4 built-in regression metrics on a simple contrived regression problem.
Note: Yourresults may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
Running the example prints the metric values at the end of each epoch.
A line plot of the 4 metrics over the training epochs is then created.
Line Plot of Built-in Keras Metrics for Regression
Note that the metrics were specified using string alias values [‘mse‘, ‘mae‘, ‘mape‘, ‘cosine‘] and were referenced as key values on the history object using their expanded function name.
We could also specify the metrics using their expanded name, as follows:
Note: Yourresults may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
Running the example reports the accuracy at the end of each training epoch.
1
2
3
4
5
6
7
8
9
10
11
...
Epoch 396/400
0s - loss: 0.5934 - acc: 0.9000
Epoch 397/400
0s - loss: 0.5932 - acc: 0.9000
Epoch 398/400
0s - loss: 0.5930 - acc: 0.9000
Epoch 399/400
0s - loss: 0.5927 - acc: 0.9000
Epoch 400/400
0s - loss: 0.5925 - acc: 0.9000
A line plot of accuracy over epoch is created.
Line Plot of Built-in Keras Metrics for Classification
Custom Metrics in Keras
You can also define your own metrics and specify the function name in the list of functions for the “metrics” argument when calling the compile() function.
A metric I often like to keep track of is Root Mean Square Error, or RMSE.
You can get an idea of how to write a custom metric by examining the code for an existing metric.
From this example and other examples of loss functions and metrics, the approach is to use standard math functions on the backend to calculate the metric of interest.
For example, we can write a custom metric to calculate RMSE as follows:
You can see the function is the same code as MSE with the addition of thesqrt() wrapping the result.
We can test this in our regression example as follows. Note that we simply list the function name directly rather than providing it as a string or alias for Keras to resolve.
Note: Yourresults may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
Running the example reports the custom RMSE metric at the end of each training epoch.
1
2
3
4
5
6
7
8
9
10
11
...
Epoch 496/500
0s - loss: 1.2992e-06 - rmse: 9.7909e-04
Epoch 497/500
0s - loss: 1.2681e-06 - rmse: 9.6731e-04
Epoch 498/500
0s - loss: 1.2377e-06 - rmse: 9.5562e-04
Epoch 499/500
0s - loss: 1.2079e-06 - rmse: 9.4403e-04
Epoch 500/500
0s - loss: 1.1788e-06 - rmse: 9.3261e-04
At the end of the run, a line plot of the custom RMSE metric is created.
Line Plot of Custom RMSE Keras Metric for Regression
Your custom metric function must operate on Keras internal data structures that may be different depending on the backend used (e.g.tensorflow.python.framework.ops.Tensor when using tensorflow) rather than the raw yhat and y values directly.
For this reason, I would recommend using the backend math functions wherever possible for consistency and execution speed.
Further Reading
This section provides more resources on the topic if you are looking go deeper.
Thanks again for another great topic on keras but I’m a R user !
I can work with keras on R, but how about to implement custom metric ‘rmse’ on keras R please ?
Because I find something like that on the github repository : metric_mean_squared_error <- function(y_true, y_pred) { keras$metrics$mean_squared_error(y_true, y_pred) } attr(metric_mean_squared_error, "py_function_name") <- "mean_squared_error"
Ok finally I make it return a value different from ‘nan’, but the result is not the same as the square root of ‘mse’ from keras ?!? Maybe due to the arg ‘axis = -1’ ?
# define base model def regression_model(): # create model model = Sequential() model.add(Dense(512, input_dim=X.shape[1], kernel_initializer=’uniform’, activation=’relu’)) model.add(Dropout(0.5)) model.add(Dense(1, kernel_initializer=’uniform’)) # compile model model.compile(loss=’mse’, optimizer=’sgd’, metrics=[RMSE]) return model
# evaluate model estimator = KerasRegressor(build_fn=regression_model, nb_epoch=100, batch_size=32, verbose=0) kfold = KFold(n_splits=3, random_state=1) reg_results = cross_val_score(estimator, X, Y, cv=kfold)
Yes, this is to be expected. Machine learning algorithms are stochastic meaning that the same algorithm on the same data will give different results each time it is run. See this post for more details: https://machinelearningmastery.com/randomness-in-machine-learning/
Dear Jason, Thank you again for the awsome blog and clear explanations If I understood well, RMSE should be equal to sqrt(mse), but this is not the case for my data: Epoch 130/1000
Hi Jason, thanks for the helpful blog. Quick question regarding your reply here, if the rmse metric is calculated at the end of each epoch, why is it constantly being updated during an epoch whenever you’re training?
Thanks for your reply. If that’s the case, why is the square root of the MSE loss function not equal to the RMSE metric value from above if they are both calculated at the end of each batch?
They should be, and if not, then there is a difference in the samples used to calculate the score – e.g. batch vs epoch, or a difference in precision between the two calculations causing rounding errors.
You could try digging into the code if this matters.
Generally, I recommend a separate standalone evaluation of model performance and only use training values as a rough/directional assessment.
I’ve been having this same problem. Readin the documentation you can see that by default, the metrics are evaluated by batch and the averaged.
“In this case, the scalar metric value you are tracking during training and evaluation is the average of the per-batch metric values for all batches see during a given epoch (or during a given call to model.evaluate()).”
Thanks for the article. How does Keras compute a mean statistic in a per batch fashion? Does it internally (magically) aggregate the sum and count to that point in the epoch and print the measure or does it compute the measure per batch and then again re-compute the metric at the end of each epoch over the entire data?
The issue is that I am trying to calculate the loss based on IoU (Intersection over union) and I have no clue how to do it using my backend (TensorFlow) My output is like this(xmin,ymin,xmax,ymax)
history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2) I thought the duration of batch is equal to one epoch, since batch_size=len(X). If it is correct?
Furthermore, it seems that the loss of epoch is also updated each iteration.
Thanks a lot for your time to explain and find the link.
I am sorry. I think I did not express my thoughts correctly.
In the above example, history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
batch_size=len(X) batch_size: Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32.
Since batch_size has been specified as the length of testset, may I consider one epoch comprises 1 batch and the end of a batch is the time when an epoch is end? Model ’mse’ loss is the rmse^2.
Thanks for the great article, Jason. I have 2 questions;
1) I have a pipeline which has a sequence like : Normalizer –> KerasRegressor Can I simply use history = pipeline.fit(..) then plot metrics ?
2) I have a KFold crossvalidation like that: kfold = StratifiedKFold(n_splits=3) results = cross_val_score(pipeline, X, Y, cv=kfold, scoring = mape) How I can plot that 3 CV fits’ metrics?
What is the different between these two lines score = model.evaluate(data2_Xscaled, data2_Yscaled, verbose=verbose) y_hat = model.predict(data2_Xscaled)
objective metric is the customized one def rmse(y_true, y_pred)
I’m using MAE as metric in a multi-class classification problem with ordered classes. Because, in my problem it is not the same to classify a record of class 5 in the class 4, than to assign it to the class 1.
When defining a custom metrics function, the y_true and y_pred are of type Tensor. If I have my own function that takes numpy arrays as input how do I convert y_true and y_pred to numpy arrays?
Hello mr Jason I have a question that have confused me for so long. For a multiple output regression problem, what does the MSE loss function compute exactly ? Thank you in advance.
Hello mr Jason For a multiple output regression problem, what does the MSE loss function compute exactly ? Is it the sum of the MSE over all the output variables, the average or something else ? Thank you in advance.
Thank you so much for your Tutorial. Nowadays I follow your twitter proposals everyday. It’s great !
I have two questions:
1) regarding sequential model in the last example; if I remove activation definition = ‘relu’, in your last code example … I got a surprising better RMSE performance values… it is suggest to me that has something to do with Regression issues that works better if we do not put activation at all in the first hide layer. Is it casual result or any profound reason?
2) using a same architectural model, which is better a Regression approach (we leave out the activation in the output layer) or a multinomial classification (we set up the appropriate ‘softmax’ as activation in the output layer), imagine for example, we analyze same problem, e.g. we have all continuos label output or any discrete multiclass label (for getting for example rounded real number by their equivalent integer number), for a serie of real number samples …I mean is there any intrinsic advantage or behavior using Regression analysis vs Multinomial classification ?
If I imagine a continuos curve for a linear regression as output of prediction vs imagine for the same problem different outputs of segments to categorize a big multi-class classification, using the same main model architecture (obviously with different units and activation at output, loss and metrics at compilation, etc) …which model will perform better the regression or the multi-class (for the same problem approach) ? My intuition tell me that multi-class it is more fine because it can focus on specific segment output (classes) of the linear regression curve (and even it has more units at the output therefore more analysis it is involved. Do not worry to much, I try in the future to experiment with some specific examples, to search for my question.
Anyway, do you like women basket? congratulations Australia won to España (-Spain it is my country -) two hours ago…
I followed all of the steps and used my own metric function successfully. I was also able to plot it.
The problem that I encountered was when I tried to load the model and the saved weights in order to use model.evaluate_generator(). I keep getting the error: “Exception has occurred: ValueError too many values to unpack (expected 2) ”
I was wondering if you know how to solve this problem.
How to extract and store the accuracy output from ‘loss’ and ‘metrics’ in the model.compile step in order to pass those float values to mlflow’s log_metric() function ?
Why is the cosine proximity value negative in this case. Should it not be positive since the dot product computed is of the same vectors it should be +1.0 right?
What would be the correct interpretation of negative value in this case? In the example you have mentioned since both the vectors were same the value we received was -1.0. When i google the meaning of it certain blogs mentioned that it means that vector are similar but in opposite directions . This does not seem a correct interpretation as both vectors are same
This post helps me again. You are the best. 🙂 I have a question when I write the custom metrics for my project. Assuming that g and v are two inputs of the model, and v in the next time step is output. There is a relationship between g and v. The loss function of the model is MSE of the output. But when it comes to the metrics, I want to define it as the MSE of predicted g and observed g. When I write a custom metric to calculate the MSE, I don’t know how to make y_true represents the observed g. Did I make it clear? Thank you in advance.
I’m trying to build my own accuracy function that checks if the output sequence is same as the true answer . For example if the true answer is ” 0.2 0.4 0.6 0.8 ”, either ” 0.4 0.6 0.8 0.2” or ”0.8 0.6 0.4 0.2 ” will be define as correct. Do you have any thoughts or recommendations?
I did not find any post in your blog specifically focus on loss function, so I submit my question under this post. There are two output features in my model. I’m not sure how the loss function works. Whether the loss function returns the sum of two calculated errors or weighted sum or some other values? MSE = MSEa + MSEb ? MSE = 0.5*MSEa + 0.5*MSEb ? If it returns the weighted sum, can I define the weight? Thank you in advance.
Hi Dr.Brownlee, I want to define custom monitor metrics such as AUC for Early Stopping and ModelCheckpoint and other callbacks for monitor options and metrics for model.compile, What do i do ?
Thanks for the tutorials. Precision and Recall metrics have been removed from the latest version of keras, they cited that the metric was misleading, do you have any idea how to create a custom precision and recall metrics?
I want to write a costume metric function. This function is PSNR (Peak signal-to-noise ratio) which is most commonly used to measure the quality of reconstruction of lossy compression codecs. My loss function is MSE.
PSNR is calculated based on the MSE results. So I was wondering if there is a way to write this PSNR function using the loss that is calculated in the fitting process. Or I should calculate MSE in the body of the function and use that information to calculate PSNR.
Is there ever a limit to number of epochs? In my data set (regression) the more epochs the better the model keeps performing… Even past 500… Is anything over 2000 epochs odd??
Why is it necessary to write axis=-1? I don’t understand what axis=-1 means here. I deleted axis=-1 from the function in my codes but it is still OK to run?
It is explicitly specifying to calculate the error across the last dimension, normally this is samples, but for encoder-decoder lstms this will be time steps.
For my thesis, I did a regression cnn in keras using the four metrics you present here to be interesting for regression.
I’ve this question with respect to the cosine_proximity-metric: if the vectors in nominator and denominator of the underlying formula are 1-dimensional vectors (=being just the real valued true and predicted label), then the metric will always resolve to 1?
– Why is its use then so interesting for regression networks, or maybe networks with multiple regressed output are intended here?
– Have you got any idea how its value could be ‘-1’ instead of ‘+1’ when both the true and predicted label are positive?
Hi! I am trying to train a recurrent neural network implemented using Keras and mean square error as loss function. Is it possible to have a loss greater than 1 and the model generated by the network to work as expected?
When i try to use a model saved using rmse as metric. During loading the model load_model(….),,, it gives the following error ValueError: Unknown metric function:rmse
What is the best metric for timeseries data? My model with MSE is either good in capturing higher signals or either fails to capture low signals.. I want a better metric which would preserve correlation and MSE together..
Please if I’ve normalized my dataset ( X and Y), with MinMaxScaler for example, and if I’m using MSE or RMSE for loss and/or for metrics, the results expected (mse and rmse) are also normalized, right?
How can I get the “real” MSE and RMSE of the original data (X and Y) denormalized?
You can get real error by inverting the transform on the predictions first, then calculating error metrics. The objects typically offer an inverse_transform() function.
Ok. Right. I did it. Thanks! One more question please…
But how about if I, let’s say, normalize X and standardize Y, or vice-versa.
When inverting the transformation on the predictions [predict(X_test) = Y_pred], which scaler should I use to get the “real” Y_pred inversely transformed?
The inverse of normalized or the inverse of standardized?
I tried using a custom loss function but always fall into errors.
Mahalanobis distance (or “generalized squared interpoint distance” for its squared value[3]) can also be defined as a dissimilarity measure between two random vectors x and y of the same distribution with the covariance matrix S. d(x,y) = square [Transpose(x-y) * Inverse(S)* (x-y)] (https://en.wikipedia.org/wiki/Mahalanobis_distance)
n_classes = 4 n_samples=800 X, y = make_classification(n_samples=n_samples, n_features=20, n_informative=4, n_redundant=0, n_classes=n_classes, n_clusters_per_class=2) y = to_categorical(y) Xtrainb, testXb, ytrainb, ytestb = train_test_split(X, y, test_size = 0.3, random_state=42)
I was developing MLPRegressor model like… #create a model nn=MLPRegressor(hidden_layer_sizes=(2, 1,),activation=’logistic’,max_iter=2000,solver=’adam’,learning_rate_init=0.1,momentum=0.7,early_stopping=True, validation_fraction=0.15,) history = nn.fit(X_train, y_train, )
how can I plot mape, r^2 and how can I predict for new samples. I was scaled my data using minmax scaler???
How can I get different components of the loss function if I am using model.train_on_batch instead of model.fit? I have seen that model.train_on_batch returns me a scalar which is just the sum of different components of loss functions? Thank you.
def sampling(args): “””Reparameterization trick by sampling fr an isotropic unit Gaussian. # Arguments: args (tensor): mean and log of variance of Q(z|X) # Returns: z (tensor): sampled latent vector “”” z_mean, z_log_var = args batch = K.shape(z_mean)[0] dim = K.int_shape(z_mean)[1] # Returns the shape of tensor or variable as a tuple of int or None entries. # by default, random_normal has mean=0 and std=1.0 epsilon = K.random_normal(shape=(batch, dim)) return z_mean + K.exp(0.5 * z_log_var) * epsilon
# use reparameterization trick to push the sampling out as input z_sampled = Lambda(sampling, output_shape=(latent_dim,), name=’z’)([z_mean_encoded, z_log_var_encoded]) # Reparameterization Trick
decoder = decoder_model() # instantiate VAE model outputs = decoder(z_sampled) # z_sampled = sampled z from [z_mean_encoded and z_log_var_encoded] vae = Model(inputs, outputs, name=’vae_mlp’)
Hi Jason, can I find the accuracy of keras regression problem? From your notes, for keras regression problem only mse,rmse,mae. Is it possible for me to find the accuracy of this method?
Hi Jason, I want to ask you how to know whether the model provide a good performance for regression? Because previously u said that we cannot know the accuracy of regression. Is it by their loss mse,mae and rmse to decide the model has the good performance? I mean if the loss of mse is below than 1, then the model are good?
i have a question in keras , i am training the keras and compiling like model .compile(loss=’binary_crossentropy’,optimizer=’Nadam’, metrics=[precision_m])
precision as metric in comiliation stage.
after all these we do model.evaluate it will give two values like loss and accuracy
if i given precision as metrics it will train based on precision right ,aftering training ,model.evaluate will return the loss and precision value ,
My question is, how can I use the history object of the model to have a line plot of the model precision at the end of each epoch? What should I use inside the bracket below?
Is it ok if I use MSE for loss function and RMSE for metric? I have heard that we should not use the same or nearly identical functions for the same model. Is this opinion right? If I must use metrics=RMSE, which loss function I should use (if MSE is not allow)? Thank you so much!
I have Sub-Classed the Metric class to create a custom precision metric. Everything looks fine; I mean there is no run-time error. But I suspect there is something wrong when I see the precision scores logging in the output.
when using proper (custom) metrics (e.g. ‘rmse’) after saving the keras model (via .save method()) when you want to load again the model (via load_model() method), it give you an error because it does not understand your own defined ‘rmse’ metric… how can we solve the keras loading?
Good question, you will need to have the function defined when loading the model and specify the function via the custom_objects argument to the load_model() function. https://keras.io/api/models/model_saving_apis/
… custom_objects: Optional dictionary mapping names (strings) to custom classes or functions to be considered during deserialization.
Perhaps the model is a bad fit for your data? Perhaps you need to use a different model configuration? Perhaps you need to use a different model? Perhaps you need to use data preparation methods? Perhaps your prediction problem is really hard?
Hello, Can I use calculated (mse mape) metrics on each epoch value to compare different LSTM models? or should I consider the values from the last epoch value? Thank you so much
