Movatterモバイル変換

Calculate Recommendation Quality Metrics

Description

Calculates recommendation quality metrics for implicit-feedbackrecommender systems (fit to user-item interactions data such as "number oftimes that a user played each song in a music service") that are based onlow-rank matrix factorization or for which predicted scores can be reduced toa dot product between user and item factors/components.

These metrics are calculated on a per-user basis, by producing a ranking of theitems according to model predictions (in descending order), ignoring the itemsthat are in the training data for each user. The items that were not consumedby the user (not present in 'X_train' and not present in 'X_test') are considered"negative" entries, while the items in 'X_test' are considered "positive" entries,and the items present in 'X_train' are ignored for these calculations.

The metrics that can be calculated by this function are:

• 'P@K' ("precision-at-k"): denotes the proportion of items among the top-Krecommended (after excluding those that were already in the training data)that can be found in the test set for that user:

  P@K = \frac{1}{k} \sum_{i=1}^k r_i \in \mathcal{T}

  This is perhaps the most intuitive and straightforward metric, but it canpresent a lot of variation between users and does not take into accountaspects such as the number of available test items or the specific ranks atwhich they are shown.

• 'TP@K' (truncated precision-at-k): a truncated or standardized versionof the precision metric, which will divide instead by the minimum between'k' and the number of test items:

  TP@K = \frac{1}{\min\{k, \mathcal{T}\}} \sum_{i=1}^k r_i \in \mathcal{T}

  Note: many papers and libraries call this the "P@K" instead. The"truncated" prefix is a non-standard nomenclature introduced here todifferentiate it from the P@K metric that is calculated by this andother libraries.

• 'R@K' ("recall-at-k"): proportion of the test items that are retrievedin the top-K recommended list. Calculation is the same as precision, but thedivision is by the number of test items instead of 'k':

  R@K = \frac{1}{|\mathcal{T}|} \sum_{i=1}^k r_i \in \mathcal{T}

• 'AP@K' ("average precision-at-k"): precision and recall look at all the itemsin the top-K equally, whereas one might want to take into account also the rankingwithin this top-K list, for which this metric comes in handy."Average Precision" tries to reflect the precisions that would be obtained atdifferent recalls:

  AP@K = \frac{1}{|\mathcal{T}|} \sum_{i=1}^k (r_i \in \mathcal{T}) \cdot P@i

  This is a metric which to some degree considers precision, recall, and rank withintop-K. Intuitively, it tries to approximate the are under a precision-recalltradeoff curve.

  The average of this metric across users is known as "Mean Average Precision"or "MAP@K".

  IMPORTANT: many authors define AP@K differently, such as dividing bythe minimum between 'k' and the number of test items instead, or as the averagefor P@1..P@K (either as-is or stopping after already retrieving all the testitems) - here, the second version is offered as different metric instead.This metric is likely to be a source of mismatches when comparing againstother libraries due to all the different defintions used by different authors.

• 'TAP@K' (truncated average precision-at-k): a truncated version of theAP@K metric, which will instead divide it by the minimum between 'k' and thenumber of test items.

  Many other papers and libraries call this the "average precision" instead.

• 'NDCG@K' (normalized discounted cumulative gain at K): a ranking metriccalculated by first determining the following:

  \sum_{i=1}^k \frac{C_i}{log_2(i+1)}

  WhereC_i denotes the confidence score for an item (taken as the valuein 'X_test' for that item), with 'i' being the item ranked at a given positionfor a given user according to the model. This metric is then standardized bydividing by the maximum achievable such score given the test data.

  Unlike the other metrics:

  • It looks not only at the presence or absence of items, but also at theirconfidence score.

  • It can handle data which contains "dislikes" in the form of negativevalues (see caveats below).

  If there are only non-negative values in 'X_test', this metric will be boundedbetween zero and one.

  A note about negative values: the NDCG metric assumes that all the values arenon-negative. This implementation however can accommodate situations in whicha very small fraction of the items have negative values, in which case:(a) it will standardize the metric by dividing by a number which does notconsider the negative values in its sum; (b) it will be set to 'NA' if thereare no positive values. Be aware however that NDCG loses some of its desirableproperties in the presence of negative values.

• 'Hit@K' ("hit-at-k"): indicates whether any of the top-K recommended itemscan be found in the test set for that user. The average across users is typicallyreferred to as the "Hit Rate".

  This is a binary metric (it is either zero or one, but can also be 'NA' whenit is not possible to calculate, just like the other metrics).

• 'RR@K' ("reciprocal-rank-at-k"): inverse rank (one divided by the rank)of the first item among the top-K recommended that is in the test data.The average across users is typically referred to as the "Mean Reciprocal Rank"or MRR.

  If none of the top-K recommended items are in the test data, will be set to zero.

• 'ROC-AUC' (area under the receiver-operating characteristic curve): see theWikipedia entryfor details. This metric considers the full ranking of itemsrather than just the top-K. It is bounded between zero and one, with a value of0.5 corresponding to a random order and a value of 1 corresponding to a perfectordering (i.e. every single positive item has a higher predicted score than everysingle negative item).

  Be aware that there are different ways of calculating AUC, with some methodshaving higher precision than others. This implementation uses a fast formulawhich implies dividing two large numbers, and as such might not be as preciseto the same number of decimals as the trapezoidal method used by e.g. scikit-learn.

• 'PR-AUC' (area under the precision-recall curve): while ROC AUC provides anoverview of the overall ranking, one is typically only interested in how well itretrieves test items within top ranks, and for this the area under theprecision-recall curve can do a better job at judging rankings, albeit the metricitself is not standardized, and under the worst possible ranking, it does notevaluate to zero.

  The metric is calculated using the fast but not-so-precise rectangular method,whose formula corresponds to the AP@K metric with K=N. Some papers and librariescall this the average of this metric the "MAP" or "Mean Average Precision" instead(without the "@K").

Metrics can be calculated for a given value of 'k' (e.g. "P@3"), or forvalues ranging from 1 to 'k' (e.g. ["P@1", "P@2", "P@3"]).

This package doesNOT cover other more specialized metrics. One mightalso want to look at other indicators such as:

• Metrics that look at the rareness of the items recommended(to evaluate so-called "serendipity").

• Metrics that look at "discoverability".

• Metrics that take into account the diversity of the ranked lists.

Usage

calc.reco.metrics(  X_train,  X_test,  A,  B,  k = 5L,  item_biases = NULL,  as_df = TRUE,  by_rows = FALSE,  sort_indices = TRUE,  precision = TRUE,  trunc_precision = FALSE,  recall = FALSE,  average_precision = TRUE,  trunc_average_precision = FALSE,  ndcg = TRUE,  hit = FALSE,  rr = FALSE,  roc_auc = FALSE,  pr_auc = FALSE,  all_metrics = FALSE,  rename_k = TRUE,  break_ties_with_noise = TRUE,  min_pos_test = 1L,  min_items_pool = 2L,  consider_cold_start = TRUE,  cumulative = FALSE,  nthreads = parallel::detectCores(),  seed = 1L)

Arguments

Details

Metrics for a given user will be set to 'NA' in the followingsituations:

• All the rankeable items have the exact same predicted score.

• One or more of the predicted scores evaluates to 'NA'/'NaN'.

• There are only negative entries (no non-zero entries in the test data).

• The number of available items to rank (between positive and negative) issmaller than the requested 'k', and the metric is not affected by the exact orderwithin the top-K items (i.e. precision, recall, hit, will be ‘NA' if there’s'k' or fewer items after discarding those from the training data).

• There are inconsistencies in the data (e.g. number of entries being greaterthan the number of columns in 'X', meaning the matrices do not constitute valid CSR).

• A user does not meet the minimum criteria set by the configurable parametersfor this function.

• There are only positive entries (i.e. the user already consumed all the items).In this case, "NDCG@K" will still be calculated, while the rest will be setto 'NA'.

The NDCG@K metric with 'cumulative=TRUE' will have lower decimal precisionthan with 'cumulative=FALSE' when using 'float32' inputs - this is extremelyunlikely to be noticeable in typical datasets and small 'k', but for large 'k'and large (absolute) values in 'X_test', it might make a difference after acouple of decimal points.

Internally, it relies on BLAS function calls, so it's recommended to useR with an optimized BLAS library such as OpenBLAS or MKL for better speed - seethis linkfor instructions on getting OpenBLAS in R for Windows(Alternatively, Microsoft's R distribution comes with MKL preinstalled).

Doing computations in float32 precision depends on the packagefloat, and as such comeswith some caveats:

• On Windows, if installing 'float' from CRAN, it will use very unoptimizedroutines which will likely result in a slowdown compared to using regulardouble (numeric) type. Getting it to use an optimized BLAS library is not assimple as substituting the Rblas DLL - see thepackage's README for details.

• On macOS, it will use static linking for 'float', thus if changing the BLASlibrary used by R, it will not change the float32 functions, and getting goodperformance out of it might require compiling it from source with '-march=native'flag.

Value

Will return the calculated metrics on a per-user basis (each usercorresponding to a row):

• If passing 'as_df=TRUE' (the default), the result will be a 'data.frame',with the columns named according to the metric they represent (e.g. "p_at_3",see below for the other names that they can take). Depending on the valuepassed for 'rename_k', the column names might end in "k" or in the numberthat was passed for 'k' (e.g "p_at_3" or "p_at_k").

  If passing 'cumulative=TRUE', they will have names ranging from 1 to 'k'.

• If passing 'as_df=FALSE', the result will be a list with entries namedaccording to each metric, with 'k' as letter rather than number ('p_at_k','tp_at_k', 'r_at_k', 'ap_at_k', 'tap_at_k', 'ndcg_at_k', 'hit_at_k', 'rr_at_k','roc_auc', 'pr_auc'), plus an additional entry with the actual 'k'.

  The values under each entry will be vectors if passing'cumulative=FALSE', or matrices (users corresponding to rows) if passing'cumulative=TRUE'.

The 'ROC-AUC' and 'PR-AUC' metrics will be named just "roc_auc" and "pr_auc",since they are calculated for the full ranked predictions without stopping at 'k'.

Examples

### (See the package vignette for a better example)library(recometrics)library(Matrix)library(MatrixExtra)### Generating random datan_users <- 10Ln_items <- 20Ln_factors <- 3Lk <- 4Lset.seed(1)UserFactors <- matrix(rnorm(n_users * n_factors), nrow=n_factors)ItemFactors <- matrix(rnorm(n_items * n_factors), nrow=n_factors)X <- Matrix::rsparsematrix(n_users, n_items, .5, repr="R")X <- abs(X)### Generating a random train-test splitdata_split <- create.reco.train.test(X, split_type="all")X_train <- data_split$X_trainX_test <- data_split$X_test### Calculating these metrics### (should be poor quality, since data is random)metrics <- calc.reco.metrics(    X_train, X_test,    UserFactors, ItemFactors,    k=k, as_df=TRUE,    nthreads=1L)print(metrics)

Create Train-Test Splits of Implicit-Feedback Data

Description

Creates train-test splits of implicit-feedback data(recorded user-item interactions) for fitting and evaluating models forrecommender systems.

These splits choose "test users" and "items for a given user" separately,offering three modes of splitting the data:

• Creating training and testing sets for each user in the data (whenpassing ‘split_type=’all'').

  This is meant for cases in which the number of users is small or the users totest have already been selected (e.g. one typically does not want to createa train-test split which would leave one item for the user in the training dataand zero in the test set, or would want to have other minimum criteria for thetest set to be usable). Typically, one would want to take only a sub-sampleof users for evaluation purposes, as calculating recommendation quality metricscan take a very long time.

• Selecting a sub-set of users for testing, for which training and testingdata splits will be generated, while leaving the remainder of users with allthe data for model fitting (when passing ‘split_type=’separated'').

  This is meant to be used for fitting a model to the remainderof the data, then generating latent factors (assuming a low-rank matrix factorizationmodel) or top-K recommendations for the test users given their training data,and evaluating these recommendations on the test data for each user (which can bedone through the functioncalc.reco.metrics).

• Selecting a sub-set of users for testing as above, but adding those users tothe training data, in which case they will be the first rows (when passing‘split_type=’joined'').

  This is meant to be used for fitting a model to all such training data, andthen evaluating the produced user factors or top-K recommendations for the testusers against the test data.

  It is recommended to use the 'separated' mode, as itis more reflective of real scenarios, but some models or libraries do not have thecapabilities for producing factors/recommendations for users which where not in thetraining data, and this option then comes in handy.

Usage

create.reco.train.test(  X,  split_type = "separated",  users_test_fraction = 0.1,  max_test_users = 10000L,  items_test_fraction = 0.3,  min_items_pool = 2L,  min_pos_test = 1L,  consider_cold_start = FALSE,  seed = 1L)

Arguments

Value

Will return a list with two to four elements depending on the requested splittype:

• If passing ‘split_type=’all'', will have elements 'X_train' and 'X_test',both of which will be sparse CSR matrices (class 'dgRMatrix' from the 'Matrix' package,which can be converted to other types through e.g. 'MatrixExtra::as.csc.matrix')with the same number of rows and columns as the 'X' that was passed as input.

• If passing ‘split_type=’separated'', will have the entries 'X_train' and 'X_test'as above (but with a number of rows corresponding to the number of selected testusers instead), plus an entry 'X_rem' which will be a CSR matrix containing thedata for the remainder of the users (those which were not selected for testing andon which the recommendation model is meant to be fitted), and an entry 'users_test'which will be an integer vector containing the indices of the users/rows in 'X'which were selected for testing. The selected test users will be in sorted order,and the remaining data will remain in sorted order with the test users removed(e.g. if there's 5 users, with the second and fifth selected for testing, then'X_train' and 'X_test' will contain rows [2,5] of 'X', while 'X_rem' will containrows [1,3,4]).

• If passing ‘split_type=’joined'', will not contain the entry 'X_rem',but instead, 'X_train' will be the concatenation of 'X_train' and 'X_rem',with ‘X_train' coming first (e.g. if there’s 5 users, with the second and fifthselected for testing, then 'X_test' will contain rows [2,5] of 'X', while'X_train' will contain rows [2,5,1,3,4], in that order).

The training and testing items for each user will not intersect, and each itemin the original 'X' data for a given test user will be assigned to either thetraining or the testing sets.