Instatistics, thek-nearest neighbors algorithm (k-NN) is anon-parametricsupervised learning method. It was first developed byEvelyn Fix andJoseph Hodges in 1951,[1] and later expanded byThomas Cover.[2] Most often, it is used forclassification, as ak-NN classifier, the output of which is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among itsk nearest neighbors (k is a positiveinteger, typically small). Ifk = 1, then the object is simply assigned to the class of that single nearest neighbor.
Thek-NN algorithm can also be generalized forregression. Ink-NN regression, also known asnearest neighbor smoothing, the output is the property value for the object. This value is the average of the values ofk nearest neighbors. Ifk = 1, then the output is simply assigned to the value of that single nearest neighbor, also known asnearest neighbor interpolation.
For both classification and regression, a useful technique can be to assign weights to the contributions of the neighbors, so that nearer neighbors contribute more to the average than distant ones. For example, a common weighting scheme consists of giving each neighbor a weight of 1/d, whered is the distance to the neighbor.[3]
The input consists of thek closest training examples in adata set. The neighbors are taken from a set of objects for which the class (fork-NN classification) or the object property value (fork-NN regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required.
A peculiarity (sometimes even a disadvantage) of thek-NN algorithm is its sensitivity to the local structure of the data.Ink-NN classification the function is only approximated locally and all computation is deferred until function evaluation. Since this algorithm relies on distance, if the features represent different physical units or come in vastly different scales, then feature-wisenormalizing of the training data can greatly improve its accuracy.[4]
Suppose we have pairs taking values in, whereY is the class label ofX, so that for (and probability distributions). Given some norm on and a point, let be a reordering of the training data such that.
The training examples are vectors in a multidimensional feature space, each with a class label. The training phase of the algorithm consists only of storing thefeature vectors and class labels of the training samples.
In the classification phase,k is a user-defined constant, and an unlabeled vector (a query or test point) is classified by assigning the label which is most frequent among thek training samples nearest to that query point.
A commonly used distance metric forcontinuous variables isEuclidean distance. For discrete variables, such as for text classification, another metric can be used, such as theoverlap metric (orHamming distance). In the context of gene expression microarray data, for example,k-NN has been employed with correlation coefficients, such as Pearson and Spearman, as a metric.[5] Often, the classification accuracy ofk-NN can be improved significantly if the distance metric is learned with specialized algorithms such aslarge margin nearest neighbor orneighborhood components analysis.
A drawback of the basic "majority voting" classification occurs when the class distribution is skewed. That is, examples of a more frequent class tend to dominate the prediction of the new example, because they tend to be common among thek nearest neighbors due to their large number.[7] One way to overcome this problem is to weight the classification, taking into account the distance from the test point to each of itsk nearest neighbors. The class (or value, in regression problems) of each of thek nearest points is multiplied by a weight proportional to the inverse of the distance from that point to the test point. Another way to overcome skew is by abstraction in data representation. For example, in aself-organizing map (SOM), each node is a representative (a center) of a cluster of similar points, regardless of their density in the original training data.k-NN can then be applied to the SOM.
The best choice ofk depends upon the data; generally, larger values ofk reduces effect of the noise on the classification,[8] but make boundaries between classes less distinct. A goodk can be selected by variousheuristic techniques (seehyperparameter optimization). The special case where the class is predicted to be the class of the closest training sample (i.e. whenk = 1) is called the nearest neighbor algorithm.
The accuracy of thek-NN algorithm can be severely degraded by the presence of noisy or irrelevant features, or if the feature scales are not consistent with their importance. Much research effort has been put intoselecting orscaling features to improve classification. A particularly popular[citation needed] approach is the use ofevolutionary algorithms to optimize feature scaling.[9] Another popular approach is to scale features by themutual information of the training data with the training classes.[citation needed]
In binary (two class) classification problems, it is helpful to choosek to be an odd number as this avoids tied votes. One popular way of choosing the empirically optimalk in this setting is via bootstrap method.[10]
The most intuitive nearest neighbour type classifier is the one nearest neighbour classifier that assigns a pointx to the class of its closest neighbour in the feature space, that is.
As the size of training data set approaches infinity, the one nearest neighbour classifier guarantees an error rate of no worse than twice theBayes error rate (the minimum achievable error rate given the distribution of the data).
Thek-nearest neighbour classifier can be viewed as assigning thek nearest neighbours a weight and all others0 weight. This can be generalised to weighted nearest neighbour classifiers. That is, where theith nearest neighbour is assigned a weight, with. An analogous result on the strong consistency of weighted nearest neighbour classifiers also holds.[11]
Let denote the weighted nearest classifier with weights. Subject to regularity conditions, which in asymptotic theory are conditional variables which require assumptions to differentiate among parameters with some criteria. On the class distributions the excess risk has the following asymptotic expansion[12]for constants and where and.
The optimal weighting scheme, that balances the two terms in the display above, is given as follows: set, for and for.
With optimal weights the dominant term in the asymptotic expansion of the excess risk is. Similar results are true when using abagged nearest neighbour classifier.
k-NN is a special case of avariable-bandwidth, kernel density "balloon" estimator with a uniformkernel.[13][14]
The naive version of the algorithm is easy to implement by computing the distances from the test example to all stored examples, but it is computationally intensive for large training sets. Using an approximatenearest neighbor search algorithm makesk-NN computationally tractable even for large data sets. Many nearest neighbor search algorithms have been proposed over the years; these generally seek to reduce the number of distance evaluations actually performed.
k-NN has some strongconsistency results. As the amount of data approaches infinity, the two-classk-NN algorithm is guaranteed to yield an error rate no worse than twice theBayes error rate (the minimum achievable error rate given the distribution of the data).[2] Various improvements to thek-NN speed are possible by using proximity graphs.[15]
For multi-classk-NN classification,Cover andHart (1967) prove an upper bound error rate ofwhere is the Bayes error rate (which is the minimal error rate possible), is the asymptotick-NN error rate, andM is the number of classes in the problem. This bound is tight in the sense that both the lower and upper bounds are achievable by some distribution.[16] For and as the Bayesian error rate approaches zero, this limit reduces to "not more than twice the Bayesian error rate".
There are many results on the error rate of thek nearest neighbour classifiers.[17] Thek-nearest neighbour classifier is strongly (that is for any joint distribution on)consistent provided diverges and converges to zero as.
Let denote thek nearest neighbour classifier based on a training set of sizen. Under certain regularity conditions, theexcess risk yields the following asymptotic expansion[12]for some constants and.
The choice offers a trade off between the two terms in the above display, for which the-nearest neighbour error converges to the Bayes error at the optimal (minimax) rate.
The K-nearest neighbor classification performance can often be significantly improved through (supervised) metric learning. Popular algorithms areneighbourhood components analysis andlarge margin nearest neighbor. Supervised metric learning algorithms use the label information to learn a newmetric orpseudo-metric.
When the input data to an algorithm is too large to be processed and it is suspected to be redundant (e.g. the same measurement in both feet and meters) then the input data will be transformed into a reduced representation set of features (also named features vector). Transforming the input data into the set of features is calledfeature extraction. If the features extracted are carefully chosen it is expected that the features set will extract the relevant information from the input data in order to perform the desired task using this reduced representation instead of the full size input. Feature extraction is performed on raw data prior to applyingk-NN algorithm on the transformed data infeature space.
An example of a typicalcomputer vision computation pipeline forface recognition usingk-NN including feature extraction and dimension reduction pre-processing steps (usually implemented withOpenCV):
For high-dimensional data (e.g., with number of dimensions more than 10)dimension reduction is usually performed prior to applying thek-NN algorithm in order to avoid the effects of thecurse of dimensionality.[18]
Thecurse of dimensionality in thek-NN context basically means thatEuclidean distance is unhelpful in high dimensions because all vectors are almost equidistant to the search query vector (imagine multiple points lying more or less on a circle with the query point at the center; the distance from the query to all data points in the search space is almost the same).
Feature extraction and dimension reduction can be combined in one step usingprincipal component analysis (PCA),linear discriminant analysis (LDA), orcanonical correlation analysis (CCA) techniques as a pre-processing step, followed by clustering byk-NN onfeature vectors in reduced-dimension space. This process is also called low-dimensionalembedding.[19]
For very-high-dimensional datasets (e.g. when performing a similarity search on live video streams, DNA data or high-dimensionaltime series) running a fastapproximatek-NN search usinglocality sensitive hashing, "random projections",[20] "sketches"[21] or other high-dimensional similarity search techniques from the VLDB toolbox might be the only feasible option.
Nearest neighbor rules in effect implicitly compute thedecision boundary. It is also possible to compute the decision boundary explicitly, and to do so efficiently, so that the computational complexity is a function of the boundary complexity.[22]
Data reduction is one of the most important problems for work with huge data sets. Usually, only some of the data points are needed for accurate classification. Those data are called theprototypes and can be found as follows:
A training example surrounded by examples of other classes is called a class outlier. Causes of class outliers include:
Class outliers withk-NN produce noise. They can be detected and separated for future analysis. Given two natural numbers,k>r>0, a training example is called a (k,r)NN class-outlier if itsk nearest neighbors include more thanr examples of other classes.
Condensed nearest neighbor (CNN, theHart algorithm) is an algorithm designed to reduce the data set fork-NN classification.[23] It selects the set of prototypesU from the training data, such that 1NN withU can classify the examples almost as accurately as 1NN does with the whole data set.
Given a training setX, CNN works iteratively:
UseU instead ofX for classification. The examples that are not prototypes are called "absorbed" points.
It is efficient to scan the training examples in order of decreasing border ratio.[24] The border ratio of a training examplex is defined as
where‖x-y‖ is the distance to the closest exampley having a different color thanx, and‖x'-y‖ is the distance fromy to its closest examplex' with the same label asx.
The border ratio is in the interval [0,1] because‖x'-y‖ never exceeds‖x-y‖. This ordering gives preference to the borders of the classes for inclusion in the set of prototypesU. A point of a different label thanx is called external tox. The calculation of the border ratio is illustrated by the figure on the right. The data points are labeled by colors: the initial point isx and its label is red. External points are blue and green. The closest tox external point isy. The closest toy red point isx'. The border ratioa(x) = ‖x'-y‖ / ‖x-y‖is the attribute of the initial pointx.
Below is an illustration of CNN in a series of figures. There are three classes (red, green and blue). Fig. 1: initially there are 60 points in each class. Fig. 2 shows the 1NN classification map: each pixel is classified by 1NN using all the data. Fig. 3 shows the 5NN classification map. White areas correspond to the unclassified regions, where 5NN voting is tied (for example, if there are two green, two red and one blue points among 5 nearest neighbors). Fig. 4 shows the reduced data set. The crosses are the class-outliers selected by the (3,2)NN rule (all the three nearest neighbors of these instances belong to other classes); the squares are the prototypes, and the empty circles are the absorbed points. The left bottom corner shows the numbers of the class-outliers, prototypes and absorbed points for all three classes. The number of prototypes varies from 15% to 20% for different classes in this example. Fig. 5 shows that the 1NN classification map with the prototypes is very similar to that with the initial data set. The figures were produced using the Mirkes applet.[24]
Ink-NN regression, also known ask-NN smoothing, thek-NN algorithm is used for estimatingcontinuous variables.[citation needed] One such algorithm uses a weighted average of thek nearest neighbors, weighted by the inverse of their distance. This algorithm works as follows:
The distance to thekth nearest neighbor can also be seen as a local density estimate and thus is also a popular outlier score inanomaly detection. The larger the distance to thek-NN, the lower the local density, the more likely the query point is an outlier.[25] Although quite simple, this outlier model, along with another classic data mining method,local outlier factor, works quite well also in comparison to more recent and more complex approaches, according to a large scale experimental analysis.[26]
Aconfusion matrix or "matching matrix" is often used as a tool to validate the accuracy ofk-NN classification. More robust statistical methods such aslikelihood-ratio test can also be applied.[how?]
{{citation}}: CS1 maint: work parameter with ISBN (link)