As promised, in this chapter, we will kick off our supervised learning journey with machine learning classification, and specifically, binary classification. The goal of the chapter is to build a movie recommendation system, which is a good starting point for learning classification from a real-life example—movie streaming service providers are already doing this, and we can do the same.

In this chapter, you will learn the fundamental concepts of classification, including what it does and its various types and applications, with a focus on solving a binary classification problem using a simple, yet powerful, algorithm, Naïve Bayes. Finally, the chapter will demonstrate how to fine-tune a model, which is an important skill that every data science or machine learning practitioner should learn.

We will go into detail on the following topics:

• Getting started with classification
• Exploring...

Movie recommendation can be framed as a machine learning classification problem. If it is predicted that you’ll like a movie because you’ve liked or watched similar movies, for example, then it will be on your recommended list; otherwise, it won’t. Let’s get started by learning the important concepts of machine learning classification.

Classification is one of the main instances of supervised learning. Given a training set of data containing observations and their associated categorical outputs, the goal of classification is to learn a general rule that correctly maps theobservations (also calledfeatures orpredictive variables) to the targetcategories (also calledlabels orclasses). Putting it another way, a trained classification model will be generated after the model learns from the features and targets of training samples, as shown in the first half ofFigure 2.1. When new or unseen data comes in, the trained...

TheNaïve Bayes classifier belongs to the family of probabilistic classifiers. It computes the probabilities of each predictivefeature (also referred to as anattribute orsignal) of the data belonging to each class in order to make a prediction of the probability distribution over all classes. Of course, from the resulting probability distribution, we can conclude the most likely class that the data sample is associated with. What Naïve Bayes does specifically, as its name indicates, is as follows:

• Bayes: As in, it maps the probability of observed input features given a possible class to the probability of the class given observed pieces of evidence based on Bayes’ theorem.
• Naïve: As in, it simplifies probability computation by assuming that predictive features are mutually independent.

I will explain Bayes’ theorem with examples in the next section.

Bayes’ theorem by example

It is important...

After calculating the movie preference example by hand, as promised, we are going to implement Naïve Bayes from scratch. After that, we will implement it using thescikit-learn package.

Implementing Naïve Bayes from scratch

Before we develop the model, let’s define the toy dataset we just worked with:

>>>import numpyas np>>>X_train = np.array([...    [0,1,1],...    [0,0,1],...    [0,0,0],...    [1,1,0]])>>>Y_train = ['Y','N','Y','Y']>>>X_test = np.array([[1,1,0]])

For the model, starting with the prior, we first group the data by label and record their indices by classes:

>>>defget_label_indices(labels):..."""...    Group samples based on their labels and return indices...    @param labels: list of labels...    @return: dict, {class1: [indices], class2: [indices]}...    ...

After the toy example, it is now time to build a movie recommender (or, more specifically, movie preference classifier) using a real dataset. We herein use a movie rating dataset (https://grouplens.org/datasets/movielens/). The movie rating data was collected by the GroupLens Research group from the MovieLens website (http://movielens.org).

For demonstration purposes, we will use the stable small dataset, MovieLens 1M Dataset (which can be downloaded fromhttps://files.grouplens.org/datasets/movielens/ml-1m.zip orhttps://grouplens.org/datasets/movielens/1m/) forml-1m.zip (size: 1 MB) file). It has around 1 million ratings, ranging from 1 to 5 with half-star increments, given by 6,040 users on 3,706 movies (last updated September 2018).

Unzip theml-1m.zip file and you will see the following four files:

• movies.dat: It contains the movie information in the format ofMovieID::Title::Genres.
• ratings.dat: It...

Beyond accuracy, there are several metrics we can use to gain more insight and avoid class imbalance effects. These are as follows:

• Confusion matrix
• Precision
• Recall
• F1 score
• The area under the curve

Aconfusion matrix summarizes testing instances by their predicted values and true values, presented as a contingency table:

Figure 2.8: Contingency table for a confusion matrix

To illustrate this, we can compute the confusion matrix of our Naïve Bayes classifier. We use theconfusion_matrix function fromscikit-learn to compute it, but it is very easy to code it ourselves:

>>>from sklearn.metricsimport confusion_matrix>>>print(confusion_matrix(Y_test, prediction, labels=[0,1]))[[ 60  47] [148 431]]

As you can see from the resulting confusion matrix, there are 47 false positive cases (where the model misinterprets a dislike as a like for a movie), and 148...

Limiting the evaluation to a single fixed set may be misleading since it’s highly dependent on the specific data points chosen for that set. We can simply avoid adopting the classification results from one fixed testing set, which we did in experiments previously. Instead, we usually apply thek-fold cross-validation technique to assess how a model will generally perform in practice.

In thek-fold cross-validation setting, the original data is first randomly divided intok equal-sized subsets, in which class proportion is often preserved. Each of thesek subsets is then successively retained as the testing set for evaluating the model. During each trial, the rest of thek -1 subsets (excluding the one-fold holdout) form the training set for driving the model. Finally, the average performance across allk trials is calculated to generate an overall result:

Figure 2.10: Diagram of 3-fold cross-validation

Statistically, the...

In this chapter, you learned about the fundamental concepts of machine learning classification, including types of classification, classification performance evaluation, cross-validation, and model tuning. You also learned about the simple, yet powerful, classifier, Naïve Bayes. We went in depth through the mechanics and implementations of Naïve Bayes with a couple of examples, the most important one being the movie recommendation project.

Binary classification using Naïve Bayes was the main talking point of this chapter. In the next chapter, we will solve ad click-through prediction using another binary classification algorithm: adecision tree.

1. As mentioned earlier, we extracted user-movie relationships only from the movie rating data where most ratings are unknown. Can you also utilize data from themovies.dat andusers.dat files?
2. Practice makes perfect—another great project to deepen your understanding could be heart disease classification. The dataset can be downloaded directly fromhttps://archive.ics.uci.edu/ml/datasets/Heart+Disease.
3. Don’t forget to fine-tune the model you obtained from Exercise 2 using the techniques you learned in this chapter. What is the best AUC it achieves?

To acknowledge the use of the MovieLens dataset in this chapter, I would like to cite the following paper:

F. Maxwell Harper and Joseph A. Konstan. 2015.The MovieLens Datasets: History and Context. ACMTransactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI:http://dx.doi.org/10.1145/2827872.