Incomputer science,online machine learning is a method ofmachine learning in which data becomes available in a sequential order and is used to update the best predictor for future data at each step, as opposed to batch learning techniques which generate the best predictor by learning on the entiretraining data set at once. Online learning is a common technique used in areas of machine learning where it is computationally infeasible to train over the entire dataset, requiring the need ofout-of-core algorithms. It is also used in situations where it is necessary for the algorithm to dynamically adapt to new patterns in the data, or when the data itself is generated as a function of time, e.g., prediction of prices in the financial international markets. Online learning algorithms may be prone tocatastrophic interference, a problem that can be addressed byincremental learning approaches.

In the setting ofsupervised learning, a function of${\displaystyle f:X\to Y}$ is to be learned, where${\displaystyle X}$ is thought of as a space of inputs and${\displaystyle Y}$ as a space of outputs, that predicts well on instances that are drawn from ajoint probability distribution${\displaystyle p(x,y)}$ on${\displaystyle X\times Y}$. In reality, the learner never knows the true distribution${\displaystyle p(x,y)}$ over instances. Instead, the learner usually has access to atraining set of examples${\displaystyle (x_1,y_1),\dots ,(x_n,y_n)}$. In this setting, theloss function is given as${\displaystyle V:Y\times Y\to \mathbb{R}}$, such that${\displaystyle V(f(x),y)}$ measures the difference between the predicted value${\displaystyle f(x)}$ and the true value${\displaystyle y}$. The ideal goal is to select a function${\displaystyle f\in \mathcal{H}}$, where${\displaystyle \mathcal{H}}$ is a space of functions called a hypothesis space, so that some notion of total loss is minimized. Depending on the type of model (statistical or adversarial), one can devise different notions of loss, which lead to different learning algorithms.

In statistical learning models, the training sample${\displaystyle (x_i,y_i)}$ are assumed to have been drawn from the true distribution${\displaystyle p(x,y)}$ and the objective is to minimize the expected "risk"$${\displaystyle I[f]=\mathbb{E}[V(f(x),y)]=\int V(f(x),y)dp(x,y).}$$A common paradigm in this situation is to estimate a function${\displaystyle \hat{f}}$ throughempirical risk minimization or regularized empirical risk minimization (usuallyTikhonov regularization). The choice of loss function here gives rise to several well-known learning algorithms such as regularizedleast squares andsupport vector machines.A purely online model in this category would learn based on just the new input${\displaystyle (x_{t+1},y_{t+1})}$, the current best predictor${\displaystyle f_t}$ and some extra stored information (which is usually expected to have storage requirements independent of training data size). For many formulations, for example nonlinearkernel methods, true online learning is not possible, though a form of hybrid online learning with recursive algorithms can be used where${\displaystyle f_{t+1}}$ is permitted to depend on${\displaystyle f_t}$ and all previous data points${\displaystyle (x_1,y_1),\dots ,(x_t,y_t)}$. In this case, the space requirements are no longer guaranteed to be constant since it requires storing all previous data points, but the solution may take less time to compute with the addition of a new data point, as compared to batch learning techniques.

A common strategy to overcome the above issues is to learn using mini-batches, which process a small batch of${\displaystyle b\ge 1}$ data points at a time, this can be considered as pseudo-online learning for${\displaystyle b}$ much smaller than the total number of training points. Mini-batch techniques are used with repeated passing over the training data to obtain optimizedout-of-core versions of machine learning algorithms, for example,stochastic gradient descent. When combined withbackpropagation, this is currently the de facto training method for trainingartificial neural networks.

The simple example of linear least squares is used to explain a variety of ideas in online learning. The ideas are general enough to be applied to other settings, for example, with other convex loss functions.

Consider the setting of supervised learning with${\displaystyle f}$ being a linear function to be learned:$${\displaystyle f(x_j)=⟨w,x_j⟩=w\cdot x_j}$$where${\displaystyle x_j\in {\mathbb{R}}^d}$ is a vector of inputs (data points) and${\displaystyle w\in {\mathbb{R}}^d}$ is a linear filter vector.The goal is to compute the filter vector${\displaystyle w}$.To this end, a square loss function$${\displaystyle V(f(x_j),y_j)=(f(x_j)-y_j{)}^2=(⟨w,x_j⟩-y_j{)}^2}$$is used to compute the vector${\displaystyle w}$ that minimizes the empirical loss$${\displaystyle I_n[w]=\sum _{j=1}^{n}V(⟨w,x_j⟩,y_j)=\sum _{j=1}^n(x_j^{\mathsf{T}}w-y_j{)}^2}$$ where$${\displaystyle y_j\in \mathbb{R}.}$$

Let${\displaystyle X}$ be the${\displaystyle i\times d}$ data matrix and${\displaystyle y\in {\mathbb{R}}^i}$ is the column vector of target values after the arrival of the first${\displaystyle i}$ data points.Assuming that the covariance matrix${\displaystyle {\mathrm{\Sigma }}_i=X^{\mathsf{T}}X}$ is invertible (otherwise it is preferential to proceed in a similar fashion with Tikhonov regularization), the best solution${\displaystyle f^{\ast }(x)=⟨w^{\ast },x⟩}$ to the linear least squares problem is given by$${\displaystyle w^{\ast }=(X^{\mathsf{T}}X{)}^{-1}{X}^{\mathsf{T}}y={\mathrm{\Sigma }}_i^{-1}\sum _{j=1}^i{x}_j{y}_j.}$$

Now, calculating the covariance matrix${\displaystyle {\mathrm{\Sigma }}_i=\sum _{j=1}^i{x}_j{x}_j^{\mathsf{T}}}$ takes time${\displaystyle O(i{d}^2)}$, inverting the${\displaystyle d\times d}$ matrix takes time${\displaystyle O(d^3)}$, while the rest of the multiplication takes time${\displaystyle O(d^2)}$, giving a total time of${\displaystyle O(i{d}^2+d^3)}$. When there are${\displaystyle n}$ total points in the dataset, to recompute the solution after the arrival of every datapoint${\displaystyle i=1,\dots ,n}$, the naive approach will have a total complexity${\displaystyle O(n^2{d}^2+n{d}^3)}$. Note that when storing the matrix${\displaystyle {\mathrm{\Sigma }}_i}$, then updating it at each step needs only adding${\displaystyle x_{i+1}{x}_{i+1}^{\mathsf{T}}}$, which takes${\displaystyle O(d^2)}$ time, reducing the total time to${\displaystyle O(n{d}^2+n{d}^3)=O(n{d}^3)}$, but with an additional storage space of${\displaystyle O(d^2)}$ to store${\displaystyle {\mathrm{\Sigma }}_i}$.^[1]

The recursive least squares (RLS) algorithm considers an online approach to the least squares problem. It can be shown that by initialising${\displaystyle w_0=0\in {\mathbb{R}}^d}$ and${\displaystyle {\mathrm{\Gamma }}_0=I\in {\mathbb{R}}^{d\times d}}$, the solution of the linear least squares problem given in the previous section can be computed by the following iteration:$${\displaystyle {\mathrm{\Gamma }}_i={\mathrm{\Gamma }}_{i-1}-\frac{{\mathrm{\Gamma }}_{i-1}{x}_i{x}_i^{\mathsf{T}}{\mathrm{\Gamma }}_{i-1}}{1+x_i^{\mathsf{T}}{\mathrm{\Gamma }}_{i-1}{x}_i}}$$$${\displaystyle w_i=w_{i-1}-{\mathrm{\Gamma }}_i{x}_i\left(x_i^{\mathsf{T}}{w}_{i-1}-y_i\right)}$$The above iteration algorithm can be proved using induction on${\displaystyle i}$.^[2] The proof also shows that${\displaystyle {\mathrm{\Gamma }}_i={\mathrm{\Sigma }}_i^{-1}}$. One can look at RLS also in the context of adaptive filters (seeRLS).

The complexity for${\displaystyle n}$ steps of this algorithm is${\displaystyle O(n{d}^2)}$, which is an order of magnitude faster than the corresponding batch learning complexity. The storage requirements at every step${\displaystyle i}$ here are to store the matrix${\displaystyle {\mathrm{\Gamma }}_i}$, which is constant at${\displaystyle O(d^2)}$. For the case when${\displaystyle {\mathrm{\Sigma }}_i}$ is not invertible, consider the regularised version of the problem loss function${\displaystyle \sum _{j=1}^n{\left(x_j^{\mathsf{T}}w-y_j\right)}^2+\lambda {\Vert w\Vert }_2^2}$. Then, it's easy to show that the same algorithm works with${\displaystyle {\mathrm{\Gamma }}_0=(I+\lambda I{)}^{-1}}$, and the iterations proceed to give${\displaystyle {\mathrm{\Gamma }}_i=({\mathrm{\Sigma }}_i+\lambda I{)}^{-1}}$.^[1]

When this$${\displaystyle w_i=w_{i-1}-{\mathrm{\Gamma }}_i{x}_i\left(x_i^{\mathsf{T}}{w}_{i-1}-y_i\right)}$$ is replaced by$${\displaystyle w_i=w_{i-1}-{\gamma }_i{x}_i\left(x_i^{\mathsf{T}}{w}_{i-1}-y_i\right)=w_{i-1}-{\gamma }_i\mathrm{\nabla }V(⟨w_{i-1},x_i⟩,y_i)}$$ or${\displaystyle {\mathrm{\Gamma }}_i\in {\mathbb{R}}^{d\times d}}$ by${\displaystyle {\gamma }_i\in \mathbb{R}}$, this becomes the stochastic gradient descent algorithm. In this case, the complexity for${\displaystyle n}$ steps of this algorithm reduces to${\displaystyle O(nd)}$. The storage requirements at every step${\displaystyle i}$ are constant at${\displaystyle O(d)}$.

However, the stepsize${\displaystyle {\gamma }_i}$ needs to be chosen carefully to solve the expected risk minimization problem, as detailed above. By choosing a decaying step size${\displaystyle {\gamma }_i\approx \frac{1}{\sqrt{i}},}$ one can prove the convergence of the average iterate${\overline{w}}_n=\frac{1}{n}\sum _{i=1}^n{w}_i$. This setting is a special case ofstochastic optimization, a well known problem in optimization.^[1]

In practice, one can perform multiple stochastic gradient passes (also called cycles or epochs) over the data. The algorithm thus obtained is called incremental gradient method and corresponds to an iteration$${\displaystyle w_i=w_{i-1}-{\gamma }_i\mathrm{\nabla }V(⟨w_{i-1},x_{t_i}⟩,y_{t_i})}$$ The main difference with the stochastic gradient method is that here a sequence${\displaystyle t_i}$ is chosen to decide which training point is visited in the${\displaystyle i}$-th step. Such a sequence can be stochastic or deterministic. The number of iterations is then decoupled to the number of points (each point can be considered more than once). The incremental gradient method can be shown to provide a minimizer to the empirical risk.^[3] Incremental techniques can be advantageous when considering objective functions made up of a sum of many terms e.g. an empirical error corresponding to a very large dataset.^[1]

Kernels can be used to extend the above algorithms to non-parametric models (or models where the parameters form an infinite dimensional space). The corresponding procedure will no longer be truly online and instead involve storing all the data points, but is still faster than the brute force method. This discussion is restricted to the case of the square loss, though it can be extended to any convex loss. It can be shown by an easy induction^[1] that if${\displaystyle X_i}$ is the data matrix and${\displaystyle w_i}$ is the output after${\displaystyle i}$ steps of theSGD algorithm, then,$${\displaystyle w_i=X_i^{\mathsf{T}}{c}_i}$$ where${\displaystyle c_i=((c_i{)}_1,(c_i{)}_2,...,(c_i{)}_i)\in {\mathbb{R}}^i}$ and the sequence${\displaystyle c_i}$ satisfies the recursion:$${\displaystyle c_0=0}$$$${\displaystyle (c_i{)}_j=(c_{i-1}{)}_j,j=1,2,...,i-1}$$ and$${\displaystyle (c_i{)}_i={\gamma }_i(y_i-\sum _{j=1}^{i-1}(c_{i-1}{)}_j⟨x_j,x_i⟩)}$$Notice that here${\displaystyle ⟨x_j,x_i⟩}$ is just the standard Kernel on${\displaystyle {\mathbb{R}}^d}$, and the predictor is of the form$${\displaystyle f_i(x)=⟨w_{i-1},x⟩=\sum _{j=1}^{i-1}(c_{i-1}{)}_j⟨x_j,x⟩.}$$

Now, if a general kernel${\displaystyle K}$ is introduced instead and let the predictor be$${\displaystyle f_i(x)=\sum _{j=1}^{i-1}(c_{i-1}{)}_{j}K(x_j,x)}$$then the same proof will also show that predictor minimising the least squares loss is obtained by changing the above recursion to$${\displaystyle (c_i{)}_i={\gamma }_i(y_i-\sum _{j=1}^{i-1}(c_{i-1}{)}_{j}K(x_j,x_i))}$$The above expression requires storing all the data for updating${\displaystyle c_i}$. The total time complexity for the recursion when evaluating for the${\displaystyle n}$-th datapoint is${\displaystyle O(n^{2}dk)}$, where${\displaystyle k}$ is the cost of evaluating the kernel on a single pair of points.^[1] Thus, the use of the kernel has allowed the movement from a finite dimensional parameter space${\displaystyle w_i\in {\mathbb{R}}^d}$ to a possibly infinite dimensional feature represented by a kernel${\displaystyle K}$ by instead performing the recursion on the space of parameters${\displaystyle c_i\in {\mathbb{R}}^i}$, whose dimension is the same as the size of the training dataset. In general, this is a consequence of therepresenter theorem.^[1]

Online convex optimization (OCO)^[4] is a general framework for decision making which leveragesconvex optimization to allow for efficient algorithms. The framework is that of repeated game playing as follows:

For${\displaystyle t=1,2,...,T}$

• Learner receives input${\displaystyle x_t}$
• Learner outputs${\displaystyle w_t}$ from a fixed convex set${\displaystyle S}$
• Nature sends back a convex loss function${\displaystyle v_t:S\to \mathbb{R}}$.
• Learner suffers loss${\displaystyle v_t(w_t)}$ and updates its model

The goal is to minimizeregret, or the difference between cumulative loss and the loss of the best fixed point${\displaystyle u\in S}$ in hindsight. As an example, consider the case of online least squares linear regression. Here, the weight vectors come from the convex set${\displaystyle S={\mathbb{R}}^d}$, and nature sends back the convex loss function${\displaystyle v_t(w)=(⟨w,x_t⟩-y_t{)}^2}$. Note here that${\displaystyle y_t}$ is implicitly sent with${\displaystyle v_t}$.

Some online prediction problems however cannot fit in the framework of OCO. For example, in online classification, the prediction domain and the loss functions are not convex. In such scenarios, two simple techniques forconvexification are used:randomisation and surrogate loss functions.^{[citation needed]}

Some simple online convex optimisation algorithms are:

The simplest learning rule to try is to select (at the current step) the hypothesis that has the least loss over all past rounds. This algorithm is called Follow the leader, and round${\displaystyle t}$ is simply given by:$${\displaystyle w_t=\underset{w\in S}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{i=1}^{t-1}{v}_i(w)}$$This method can thus be looked as agreedy algorithm. For the case of online quadratic optimization (where the loss function is${\displaystyle v_t(w)={\Vert w-x_t\Vert }_2^2}$), one can show a regret bound that grows as${\displaystyle \log (T)}$. However, similar bounds cannot be obtained for the FTL algorithm for other important families of models like online linear optimization. To do so, one modifies FTL by adding regularisation.

This is a natural modification of FTL that is used to stabilise the FTL solutions and obtain better regret bounds. A regularisation function${\displaystyle R:S\to \mathbb{R}}$ is chosen and learning performed in roundt as follows:$${\displaystyle w_t=\underset{w\in S}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{i=1}^{t-1}{v}_i(w)+R(w)}$$As a special example, consider the case of online linear optimisation i.e. where nature sends back loss functions of the form${\displaystyle v_t(w)=⟨w,z_t⟩}$. Also, let${\displaystyle S={\mathbb{R}}^d}$. Suppose the regularisation function$R(w)=\frac{1}{2\eta }{\Vert w\Vert }_2^2$ is chosen for some positive number${\displaystyle \eta }$. Then, one can show that the regret minimising iteration becomes$${\displaystyle w_{t+1}=-\eta \sum _{i=1}^t{z}_i=w_t-\eta z_t}$$Note that this can be rewritten as${\displaystyle w_{t+1}=w_t-\eta \mathrm{\nabla }{v}_t(w_t)}$, which looks exactly like online gradient descent.

IfS is instead some convex subspace of${\displaystyle {\mathbb{R}}^d}$,S would need to be projected onto, leading to the modified update rule$${\displaystyle w_{t+1}={\mathrm{\Pi }}_S(-\eta \sum _{i=1}^t{z}_i)={\mathrm{\Pi }}_S(\eta {\theta }_{t+1})}$$This algorithm is known as lazy projection, as the vector${\displaystyle {\theta }_{t+1}}$ accumulates the gradients. It is also known as Nesterov's dual averaging algorithm. In this scenario of linear loss functions and quadratic regularisation, the regret is bounded by${\displaystyle O(\sqrt{T})}$, and thus the average regret goes to0 as desired.

The above proved a regret bound for linear loss functions${\displaystyle v_t(w)=⟨w,z_t⟩}$. To generalise the algorithm to any convex loss function, thesubgradient${\displaystyle \mathrm{\partial }{v}_t(w_t)}$ of${\displaystyle v_t}$ is used as a linear approximation to${\displaystyle v_t}$ near${\displaystyle w_t}$, leading to the online subgradient descent algorithm:

Initialise parameter${\displaystyle \eta ,w_1=0}$

For${\displaystyle t=1,2,...,T}$

• Predict using${\displaystyle w_t}$, receive${\displaystyle f_t}$ from nature.
• Choose${\displaystyle z_t\in \mathrm{\partial }{v}_t(w_t)}$
• If${\displaystyle S={\mathbb{R}}^d}$, update as${\displaystyle w_{t+1}=w_t-\eta z_t}$
• If${\displaystyle S\subset {\mathbb{R}}^d}$, project cumulative gradients onto${\displaystyle S}$ i.e.${\displaystyle w_{t+1}={\mathrm{\Pi }}_S(\eta {\theta }_{t+1}),{\theta }_{t+1}={\theta }_t+z_t}$

One can use the OSD algorithm to derive${\displaystyle O(\sqrt{T})}$ regret bounds for the online version ofSVM's for classification, which use thehinge loss${\displaystyle v_t(w)=max\{0,1-y_t(w\cdot x_t)\}}$

Quadratically regularised FTRL algorithms lead to lazily projected gradient algorithms as described above. To use the above for arbitrary convex functions and regularisers, one usesonline mirror descent. The optimal regularization in hindsight can be derived for linear loss functions, this leads to theAdaGrad algorithm. For the Euclidean regularisation, one can show a regret bound of${\displaystyle O(\sqrt{T})}$, which can be improved further to a${\displaystyle O(\log T)}$ for strongly convex and exp-concave loss functions.

Continual learning means constantly improving the learned model by processing continuous streams of information.^[5] Continual learning capabilities are essential for software systems and autonomous agents interacting in an ever changing real world. However, continual learning is a challenge for machine learning and neural network models since the continual acquisition of incrementally available information from non-stationary data distributions generally leads tocatastrophic forgetting.

The paradigm of online learning has different interpretations depending on the choice of the learning model, each of which has distinct implications about the predictive quality of the sequence of functions${\displaystyle f_1,f_2,\dots ,f_n}$. The prototypical stochastic gradient descent algorithm is used for this discussion. As noted above, its recursion is given by$${\displaystyle w_t=w_{t-1}-{\gamma }_t\mathrm{\nabla }V(⟨w_{t-1},x_t⟩,y_t)}$$

The first interpretation consider thestochastic gradient descent method as applied to the problem of minimizing the expected risk${\displaystyle I[w]}$ defined above.^[6] Indeed, in the case of an infinite stream of data, since the examples${\displaystyle (x_1,y_1),(x_2,y_2),\dots }$ are assumed to be drawn i.i.d. from the distribution${\displaystyle p(x,y)}$, the sequence of gradients of${\displaystyle V(\cdot ,\cdot )}$ in the above iteration are an i.i.d. sample of stochastic estimates of the gradient of the expected risk${\displaystyle I[w]}$ and therefore one can apply complexity results for the stochastic gradient descent method to bound the deviation${\displaystyle I[w_t]-I[w^{\ast }]}$, where${\displaystyle w^{\ast }}$ is the minimizer of${\displaystyle I[w]}$.^[7] This interpretation is also valid in the case of a finite training set; although with multiple passes through the data the gradients are no longer independent, still complexity results can be obtained in special cases.

The second interpretation applies to the case of a finite training set and considers the SGD algorithm as an instance of incremental gradient descent method.^[3] In this case, one instead looks at the empirical risk:$${\displaystyle I_n[w]=\frac{1}{n}\sum _{i=1}^{n}V(⟨w,x_i⟩,y_i).}$$Since the gradients of${\displaystyle V(\cdot ,\cdot )}$ in the incremental gradient descent iterations are also stochastic estimates of the gradient of${\displaystyle I_n[w]}$, this interpretation is also related to the stochastic gradient descent method, but applied to minimize the empirical risk as opposed to the expected risk. Since this interpretation concerns the empirical risk and not the expected risk, multiple passes through the data are readily allowed and actually lead to tighter bounds on the deviations${\displaystyle I_n[w_t]-I_n[w_n^{\ast }]}$, where${\displaystyle w_n^{\ast }}$ is the minimizer of${\displaystyle I_n[w]}$.

• Vowpal Wabbit: Open-source fast out-of-core online learning system which is notable for supporting a number of machine learningreductions, importance weighting and a selection of different loss functions and optimisation algorithms. It uses thehashing trick for bounding the size of the set of features independent of the amount of training data.
• scikit-learn: Provides out-of-core implementations of algorithms for
  • Classification:Perceptron,SGD classifier,Naive bayes classifier.
  • Regression: SGD Regressor, Passive Aggressive regressor.
  • Clustering:Mini-batch k-means.
  • Feature extraction:Mini-batch dictionary learning,Incremental PCA.

Learning paradigms

• Incremental learning
• Lazy learning
• Offline learning, the opposite model
• Reinforcement learning
• Multi-armed bandit
• Supervised learning

General algorithms

• Online algorithm
• Online optimization
• Streaming algorithm
• Stochastic gradient descent

Learning models

• Adaptive Resonance Theory
• Hierarchical temporal memory
• k-nearest neighbor algorithm
• Learning vector quantization
• Perceptron

• 6.883: Online Methods in Machine Learning: Theory and Applications. Alexander Rakhlin. MIT

• Online algorithms
• Machine learning algorithms

• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from September 2019