In the theory ofstochastic processes, theKarhunen–Loève theorem (named afterKari Karhunen andMichel Loève), also known as theKosambi–Karhunen–Loève theorem^[1][2] states that astochastic process can be represented as an infinitelinear combination oforthogonal functions, analogous to aFourier series representation of a function on a bounded interval. The transformation is also known asHotelling transform andeigenvector transform, and is closely related toprincipal component analysis (PCA) technique widely used in image processing and in data analysis in many fields.^[3]

There exist many such expansions of a stochastic process: if the process is indexed over[a,b], anyorthonormal basis ofL²([a,b]) yields an expansion thereof in that form. The importance of the Karhunen–Loève theorem is that it yields the best such basis in the sense that it minimizes the totalmean squared error.

In contrast to a Fourier series where the coefficients are fixed numbers and the expansion basis consists ofsinusoidal functions (that is,sine andcosine functions), the coefficients in the Karhunen–Loève theorem arerandom variables and the expansion basis depends on the process. In fact, the orthogonal basis functions used in this representation are determined by thecovariance function of the process. One can think that the Karhunen–Loève transform adapts to the process in order to produce the best possible basis for its expansion.

In the case of acentered stochastic process{X_t}_{t ∈ [a,b]} (centered meansE[X_t] = 0 for allt ∈ [a,b]) satisfying a technical continuity condition,X admits a decomposition

${\displaystyle X_t=\sum _{k=1}^{\mathrm{\infty }}{Z}_k{e}_k(t)}$

whereZ_k are pairwiseuncorrelated random variables and the functionse_k are continuous real-valued functions on[a,b] that are pairwiseorthogonal inL²([a,b]). It is therefore sometimes said that the expansion isbi-orthogonal since the random coefficientsZ_k are orthogonal in the probability space while the deterministic functionse_k are orthogonal in the time domain. The general case of a processX_t that is not centered can be brought back to the case of a centered process by consideringX_t −E[X_t] which is a centered process.

Moreover, if the process isGaussian, then the random variablesZ_k are Gaussian andstochastically independent. This result generalizes theKarhunen–Loève transform. An important example of a centered real stochastic process on[0, 1] is theWiener process; the Karhunen–Loève theorem can be used to provide a canonical orthogonal representation for it. In this case the expansion consists of sinusoidal functions.

The above expansion into uncorrelated random variables is also known as theKarhunen–Loève expansion orKarhunen–Loève decomposition. Theempirical version (i.e., with the coefficients computed from a sample) is known as theKarhunen–Loève transform (KLT),principal component analysis,proper orthogonal decomposition (POD),empirical orthogonal functions (a term used inmeteorology andgeophysics), or theHotelling transform.

• Throughout this article, we will consider a random processX_t defined over aprobability space(Ω,F,P) and indexed over a closed interval[a,b], which issquare-integrable, has zero-mean, and with covariance functionK_X(s,t). In other words, we have:

${\displaystyle \mathrm{\forall }t\in [a,b]\mathbf{E}[X_t]=0,}$

${\displaystyle \mathrm{\forall }t,s\in [a,b]K_X(s,t)=\mathbf{E}[X_s{X}_t].}$

The square-integrable condition is logically equivalent to${\displaystyle K_X(s,t)}$ being finite for all${\displaystyle s,t\in [a,b]}$.^[4]

• We associate toK_X alinear operator (more specifically aHilbert–Schmidt integral operator)T_KX defined in the following way:

SinceT_KX is a linear endomorphism, it makes sense to talk about its eigenvaluesλ_k and eigenfunctionse_k, which are found by solving the homogeneous Fredholmintegral equation of the second kind

${\displaystyle {\int }_a^b{K}_X(s,t)e_k(s)ds={\lambda }_k{e}_k(t)}$.

Theorem. LetX_t be a zero-mean square-integrable stochastic process defined over a probability space(Ω,F,P) and indexed over a closed and bounded interval [a, b], with continuous covariance functionK_X(s,t).

ThenK_X(s,t) is aMercer kernel and lettinge_k be an orthonormal basis onL²([a,b]) formed by the eigenfunctions ofT_KX with respective eigenvaluesλ_k, X_t admits the following representation

${\displaystyle X_t=\sum _{k=1}^{\mathrm{\infty }}{Z}_k{e}_k(t)}$

where the convergence is inL², uniform int and

${\displaystyle Z_k={\int }_a^b{X}_t{e}_k(t)dt}$

Furthermore, the random variablesZ_k have zero-mean, are uncorrelated and have varianceλ_k

${\displaystyle \mathbf{E}[Z_k]=0,\mathrm{\forall }k\in \mathbb{N}\text{and}\mathbf{E}[Z_i{Z}_j]={\delta }_{ij}{\lambda }_j,\mathrm{\forall }i,j\in \mathbb{N}}$

Note that by generalizations of Mercer's theorem we can replace the interval [a,b] with other compact spacesC and theLebesgue measure on [a,b] with aBorel measure whose support isC.

• The covariance functionK_X satisfies the definition of a Mercer kernel. ByMercer's theorem, there consequently exists a setλ_k,e_k(t) of eigenvalues and eigenfunctions ofT_KX forming an orthonormal basis ofL²([a,b]), andK_X can be expressed as

${\displaystyle K_X(s,t)=\sum _{k=1}^{\mathrm{\infty }}{\lambda }_k{e}_k(s)e_k(t)}$

• The processX_t can be expanded in terms of the eigenfunctionse_k as:

${\displaystyle X_t=\sum _{k=1}^{\mathrm{\infty }}{Z}_k{e}_k(t)}$

where the coefficients (random variables)Z_k are given by the projection ofX_t on the respective eigenfunctions

${\displaystyle Z_k={\int }_a^b{X}_t{e}_k(t)dt}$

• We may then derive

where we have used the fact that thee_k are eigenfunctions ofT_KX and are orthonormal.

• Let us now show that the convergence is inL². Let

${\displaystyle S_N=\sum _{k=1}^N{Z}_k{e}_k(t).}$

Then:

which goes to 0 by Mercer's theorem.

Since the limit in the mean of jointly Gaussian random variables is jointly Gaussian, and jointly Gaussian random (centered) variables are independentif and only if they are orthogonal, we can also conclude:

Theorem. The variablesZ_i have a joint Gaussian distribution and are stochastically independent if the original process{X_t}_t is Gaussian.

In the Gaussian case, since the variablesZ_i are independent, we can say more:

${\displaystyle \underset{N\to \mathrm{\infty }}{lim}\sum _{i=1}^N{e}_i(t)Z_i(\omega )=X_t(\omega )}$

almost surely.

This is a consequence of the independence of theZ_k.

In the introduction, we mentioned that the truncated Karhunen–Loeve expansion was the best approximation of the original process in the sense that it reduces the total mean-square error resulting of its truncation. Because of this property, it is often said that the KL transform optimally compacts the energy.

More specifically, given any orthonormal basis{f_k} ofL²([a,b]), we may decompose the processX_t as:

${\displaystyle X_t(\omega )=\sum _{k=1}^{\mathrm{\infty }}{A}_k(\omega )f_k(t)}$

where

${\displaystyle A_k(\omega )={\int }_a^b{X}_t(\omega )f_k(t)dt}$

and we may approximateX_t by the finite sum

${\displaystyle {\hat{X}}_t(\omega )=\sum _{k=1}^N{A}_k(\omega )f_k(t)}$

for some integerN.

Claim. Of all such approximations, the KL approximation is the one that minimizes the total mean square error (provided we have arranged the eigenvalues in decreasing order).

An important observation is that since the random coefficientsZ_k of the KL expansion are uncorrelated, theBienaymé formula asserts that the variance ofX_t is simply the sum of the variances of the individual components of the sum:

${\displaystyle \mathrm{var}[X_t]=\sum _{k=0}^{\mathrm{\infty }}{e}_k(t{)}^2\mathrm{var}[Z_k]=\sum _{k=1}^{\mathrm{\infty }}{\lambda }_k{e}_k(t{)}^2}$

Integrating over [a,b] and using the orthonormality of thee_k, we obtain that the total variance of the process is:

${\displaystyle {\int }_a^b\mathrm{var}[X_t]dt=\sum _{k=1}^{\mathrm{\infty }}{\lambda }_k}$

In particular, the total variance of theN-truncated approximation is

${\displaystyle \sum _{k=1}^N{\lambda }_k.}$

As a result, theN-truncated expansion explains

${\displaystyle \frac{\sum _{k=1}^N{\lambda }_k}{\sum _{k=1}^{\mathrm{\infty }}{\lambda }_k}}$

of the variance; and if we are content with an approximation that explains, say, 95% of the variance, then we just have to determine an${\displaystyle N\in \mathbb{N}}$ such that

${\displaystyle \frac{\sum _{k=1}^N{\lambda }_k}{\sum _{k=1}^{\mathrm{\infty }}{\lambda }_k}\ge \mathrm{0.95.}}$

Given a representation of${\displaystyle X_t=\sum _{k=1}^{\mathrm{\infty }}{W}_k{\phi }_k(t)}$, for some orthonormal basis${\displaystyle {\phi }_k(t)}$ and random${\displaystyle W_k}$, we let${\displaystyle p_k=\mathbb{E}[|W_k{|}^2]/\mathbb{E}[|X_t{|}_{L^2}^2]}$, so that${\displaystyle \sum _{k=1}^{\mathrm{\infty }}{p}_k=1}$. We may then define the representationentropy to be${\displaystyle H(\{{\phi }_k\})=-\sum _i{p}_k\log (p_k)}$. Then we have${\displaystyle H(\{{\phi }_k\})\ge H(\{e_k\})}$, for all choices of${\displaystyle {\phi }_k}$. That is, the KL-expansion has minimal representation entropy.

Proof:

Denote the coefficients obtained for the basis${\displaystyle e_k(t)}$ as${\displaystyle p_k}$, and for${\displaystyle {\phi }_k(t)}$ as${\displaystyle q_k}$.

Choose${\displaystyle N\ge 1}$. Note that since${\displaystyle e_k}$ minimizes the mean squared error, we have that

${\displaystyle \mathbb{E}{\left|\sum _{k=1}^N{Z}_k{e}_k(t)-X_t\right|}_{L^2}^2\le \mathbb{E}{\left|\sum _{k=1}^N{W}_k{\phi }_k(t)-X_t\right|}_{L^2}^2}$

Expanding the right hand size, we get:

${\displaystyle \mathbb{E}{\left|\sum _{k=1}^N{W}_k{\phi }_k(t)-X_t\right|}_{L^2}^2=\mathbb{E}|X_t^2{|}_{L^2}+\sum _{k=1}^N\sum _{\ell =1}^N\mathbb{E}[W_{\ell }{\phi }_{\ell }(t)W_k^{\ast }{\phi }_k^{\ast }(t){]}_{L^2}-\sum _{k=1}^N\mathbb{E}[W_k{\phi }_k{X}_t^{\ast }{]}_{L^2}-\sum _{k=1}^N\mathbb{E}[X_t{W}_k^{\ast }{\phi }_k^{\ast }(t){]}_{L^2}}$

Using the orthonormality of${\displaystyle {\phi }_k(t)}$, and expanding${\displaystyle X_t}$ in the${\displaystyle {\phi }_k(t)}$ basis, we get that the right hand size is equal to:

${\displaystyle \mathbb{E}[X_t{]}_{L^2}^2-\sum _{k=1}^N\mathbb{E}[|W_k{|}^2]}$

We may perform identical analysis for the${\displaystyle e_k(t)}$, and so rewrite the above inequality as:

${\displaystyle {\displaystyle \mathbb{E}[X_t{]}_{L^2}^2-\sum _{k=1}^N\mathbb{E}[|Z_k{|}^2]}\le {\displaystyle \mathbb{E}[X_t{]}_{L^2}^2-\sum _{k=1}^N\mathbb{E}[|W_k{|}^2]}}$

Subtracting the common first term, and dividing by${\displaystyle \mathbb{E}[|X_t{|}_{L^2}^2]}$, we obtain that:

${\displaystyle \sum _{k=1}^N{p}_k\ge \sum _{k=1}^N{q}_k}$

This implies that:

${\displaystyle -\sum _{k=1}^{\mathrm{\infty }}{p}_k\log (p_k)\le -\sum _{k=1}^{\mathrm{\infty }}{q}_k\log (q_k)}$

Consider a whole class of signals we want to approximate over the firstM vectors of a basis. These signals are modeled as realizations of a random vectorY[n] of sizeN. To optimize the approximation we design a basis that minimizes the averageapproximation error. This section proves that optimal bases are Karhunen–Loeve bases that diagonalize the covariance matrix ofY. The random vectorY can be decomposed in an orthogonal basis

${\displaystyle {\left\{g_m\right\}}_{0\le m\le N}}$

as follows:

${\displaystyle Y=\sum _{m=0}^{N-1}⟨Y,g_m⟩g_m,}$

where each

${\displaystyle ⟨Y,g_m⟩=\sum _{n=0}^{N-1}Y[n]g_m^{\ast }[n]}$

is a random variable. The approximation from the firstM ≤N vectors of the basis is

${\displaystyle Y_M=\sum _{m=0}^{M-1}⟨Y,g_m⟩g_m}$

The energy conservation in an orthogonal basis implies

${\displaystyle \epsilon [M]=\mathbf{E}\left\{{\Vert Y-Y_M\Vert }^2\right\}=\sum _{m=M}^{N-1}\mathbf{E}\left\{{\left|⟨Y,g_m⟩\right|}^2\right\}}$

This error is related to the covariance ofY defined by

${\displaystyle R[n,m]=\mathbf{E}\left\{Y[n]Y^{\ast }[m]\right\}}$

For any vectorx[n] we denote byK thecovariance operator represented by this matrix,

${\displaystyle \mathbf{E}\left\{{\left|⟨Y,x⟩\right|}^2\right\}=⟨Kx,x⟩=\sum _{n=0}^{N-1}\sum _{m=0}^{N-1}R[n,m]x[n]x^{\ast }[m]}$

The errorε[M] is therefore a sum of the lastN −M coefficients of the covariance operator

${\displaystyle \epsilon [M]=\sum _{m=M}^{N-1}⟨K{g}_m,g_m⟩}$

The covariance operatorK is Hermitian and Positive and is thus diagonalized in an orthogonal basis called a Karhunen–Loève basis. The following theorem states that a Karhunen–Loève basis is optimal for linear approximations.

Theorem (Optimality of Karhunen–Loève basis). LetK be a covariance operator. For allM ≥ 1, the approximation error

${\displaystyle \epsilon [M]=\sum _{m=M}^{N-1}⟨K{g}_m,g_m⟩}$

is minimum if and only if

is a Karhunen–Loeve basis ordered by decreasing eigenvalues.

Linear approximations project the signal onM vectors a priori. The approximation can be made more precise by choosing theM orthogonal vectors depending on the signal properties. This section analyzes the general performance of these non-linear approximations. A signal${\displaystyle f\in \mathrm{H}}$ is approximated with M vectors selected adaptively in an orthonormal basis for${\displaystyle \mathrm{H}}$^{[definition needed]}

${\displaystyle \mathrm{B}={\left\{g_m\right\}}_{m\in \mathbb{N}}}$

Let${\displaystyle f_M}$ be the projection of f over M vectors whose indices are inI_M:

${\displaystyle f_M=\sum _{m\in I_M}⟨f,g_m⟩g_m}$

The approximation error is the sum of the remaining coefficients

${\displaystyle \epsilon [M]=\left\{{\Vert f-f_M\Vert }^2\right\}=\sum _{m\notin I_M}^{N-1}\left\{{\left|⟨f,g_m⟩\right|}^2\right\}}$

To minimize this error, the indices inI_M must correspond to the M vectors having the largest inner product amplitude

${\displaystyle \left|⟨f,g_m⟩\right|.}$

These are the vectors that best correlate f. They can thus be interpreted as the main features of f. The resulting error is necessarily smaller than the error of alinear approximation which selects the M approximation vectors independently of f. Let us sort

${\displaystyle {\left\{\left|⟨f,g_m⟩\right|\right\}}_{m\in \mathbb{N}}}$

in decreasing order

${\displaystyle \left|⟨f,g_{m_k}⟩\right|\ge \left|⟨f,g_{m_{k+1}}⟩\right|.}$

The best non-linear approximation is

${\displaystyle f_M=\sum _{k=1}^M⟨f,g_{m_k}⟩g_{m_k}}$

It can also be written as inner product thresholding:

${\displaystyle f_M=\sum _{m=0}^{\mathrm{\infty }}{\theta }_T\left(⟨f,g_m⟩\right)g_m}$

with

The non-linear error is

${\displaystyle \epsilon [M]=\left\{{\Vert f-f_M\Vert }^2\right\}=\sum _{k=M+1}^{\mathrm{\infty }}\left\{{\left|⟨f,g_{m_k}⟩\right|}^2\right\}}$

this error goes quickly to zero as M increases, if the sorted values of${\displaystyle \left|⟨f,g_{m_k}⟩\right|}$ have a fast decay as k increases. This decay is quantified by computing the${\displaystyle {\mathrm{I}}^{\mathrm{P}}}$ norm of the signal inner products in B:

${\displaystyle \Vert f{\Vert }_{\mathrm{B},p}={\left(\sum _{m=0}^{\mathrm{\infty }}{\left|⟨f,g_m⟩\right|}^p\right)}^{\frac{1}{p}}}$

The following theorem relates the decay ofε[M] to${\displaystyle \Vert f{\Vert }_{\mathrm{B},p}}$

Theorem (decay of error). If withp < 2 then

${\displaystyle \epsilon [M]\le \frac{\Vert f{\Vert }_{\mathrm{B},p}^2}{\frac{2}{p}-1}{M}^{1-\frac{2}{p}}}$

and

${\displaystyle \epsilon [M]=o\left(M^{1-\frac{2}{p}}\right).}$

Conversely, if${\displaystyle \epsilon [M]=o\left(M^{1-\frac{2}{p}}\right)}$ then

for anyq >p.

To further illustrate the differences between linear and non-linear approximations, we study the decomposition of a simple non-Gaussian random vector in a Karhunen–Loève basis. Processes whose realizations have a random translation are stationary. The Karhunen–Loève basis is then a Fourier basis and we study its performance. To simplify the analysis, consider a random vectorY[n] of sizeN that is random shift moduloN of a deterministic signalf[n] of zero mean

${\displaystyle \sum _{n=0}^{N-1}f[n]=0}$

${\displaystyle Y[n]=f[(n-p)modN]}$

The random shiftP is uniformly distributed on [0, N − 1]:

Clearly

${\displaystyle \mathbf{E}\{Y[n]\}=\frac{1}{N}\sum _{p=0}^{N-1}f[(n-p)modN]=0}$

and

${\displaystyle R[n,k]=\mathbf{E}\{Y[n]Y[k]\}=\frac{1}{N}\sum _{p=0}^{N-1}f[(n-p)modN]f[(k-p)modN]=\frac{1}{N}f\mathrm{\Theta }\overline{f}[n-k],\overline{f}[n]=f[-n]}$

Hence

${\displaystyle R[n,k]=R_Y[n-k],R_Y[k]=\frac{1}{N}f\mathrm{\Theta }\overline{f}[k]}$

Since R_Y is N periodic, Y is a circular stationary random vector. The covariance operator is acircular convolution with R_Y and is therefore diagonalized in the discrete Fourier Karhunen–Loève basis

The power spectrum is Fourier transform ofR_Y:

${\displaystyle P_Y[m]={\hat{R}}_Y[m]=\frac{1}{N}{\left|\hat{f}[m]\right|}^2}$

Example: Consider an extreme case where${\displaystyle f[n]=\delta [n]-\delta [n-1]}$. A theorem stated above guarantees that the Fourier Karhunen–Loève basis produces a smaller expected approximation error than a canonical basis of Diracs. Indeed, we do not know a priori the abscissa of the non-zero coefficients ofY, so there is no particular Dirac that is better adapted to perform the approximation. But the Fourier vectors cover the whole support of Y and thus absorb a part of the signal energy.

${\displaystyle \mathbf{E}\left\{{\left|⟨Y[n],\frac{1}{\sqrt{N}}{e}^{i2\pi mn/N}⟩\right|}^2\right\}=P_Y[m]=\frac{4}{N}{\sin }^2\left(\frac{\pi k}{N}\right)}$

Selecting higher frequency Fourier coefficients yields a better mean-square approximation than choosing a priori a few Dirac vectors to perform the approximation. The situation is totally different for non-linear approximations. If${\displaystyle f[n]=\delta [n]-\delta [n-1]}$ then the discrete Fourier basis is extremely inefficient because f and hence Y have an energy that is almost uniformly spread among all Fourier vectors. In contrast, since f has only two non-zero coefficients in the Dirac basis, a non-linear approximation of Y withM ≥ 2 gives zero error.^[5]

We have established the Karhunen–Loève theorem and derived a few properties thereof. We also noted that one hurdle in its application was the numerical cost of determining the eigenvalues and eigenfunctions of its covariance operator through the Fredholm integral equation of the second kind

${\displaystyle {\int }_a^b{K}_X(s,t)e_k(s)ds={\lambda }_k{e}_k(t).}$

However, when applied to a discrete and finite process${\displaystyle {\left(X_n\right)}_{n\in \{1,\dots ,N\}}}$, the problem takes a much simpler form and standard algebra can be used to carry out the calculations.

Note that a continuous process can also be sampled atN points in time in order to reduce the problem to a finite version.

We henceforth consider a randomN-dimensional vector${\displaystyle X={\left(X_1{X}_2\dots X_N\right)}^T}$. As mentioned above,X could containN samples of a signal but it can hold many more representations depending on the field of application. For instance it could be the answers to a survey or economic data in an econometrics analysis.

As in the continuous version, we assume thatX is centered, otherwise we can let${\displaystyle X:=X-{\mu }_X}$ (where${\displaystyle {\mu }_X}$ is themean vector ofX) which is centered.

Let us adapt the procedure to the discrete case.

Recall that the main implication and difficulty of the KL transformation is computing the eigenvectors of the linear operator associated to the covariance function, which are given by the solutions to the integral equation written above.

Define Σ, the covariance matrix ofX, as anN ×N matrix whose elements are given by:

${\displaystyle {\mathrm{\Sigma }}_{ij}=\mathbf{E}[X_i{X}_j],\mathrm{\forall }i,j\in \{1,\dots ,N\}}$

Rewriting the above integral equation to suit the discrete case, we observe that it turns into:

${\displaystyle \sum _{j=1}^N{\mathrm{\Sigma }}_{ij}{e}_j=\lambda e_i\iff \mathrm{\Sigma }e=\lambda e}$

where${\displaystyle e=(e_1{e}_2\dots e_N{)}^T}$ is anN-dimensional vector.

The integral equation thus reduces to a simple matrix eigenvalue problem, which explains why the PCA has such a broad domain of applications.

Since Σ is a positive definite symmetric matrix, it possesses a set of orthonormal eigenvectors forming a basis of${\displaystyle {\mathbb{R}}^N}$, and we write${\displaystyle \{{\lambda }_i,{\phi }_i{\}}_{i\in \{1,\dots ,N\}}}$ this set of eigenvalues and corresponding eigenvectors, listed in decreasing values ofλ_i. Let alsoΦ be the orthonormal matrix consisting of these eigenvectors:

It remains to perform the actual KL transformation, called theprincipal component transform in this case. Recall that the transform was found by expanding the process with respect to the basis spanned by the eigenvectors of the covariance function. In this case, we hence have:

${\displaystyle X=\sum _{i=1}^N⟨{\phi }_i,X⟩{\phi }_i=\sum _{i=1}^N{\phi }_i^{T}X{\phi }_i}$

In a more compact form, the principal component transform ofX is defined by:

${\displaystyle \{\begin{array}{l}Y={\mathrm{\Phi }}^{T}X\\ X=\mathrm{\Phi }Y\end{array}}$

Thei-th component ofY is${\displaystyle Y_i={\phi }_i^{T}X}$, the projection ofX on${\displaystyle {\phi }_i}$ and the inverse transformX = ΦY yields the expansion ofX on the space spanned by the${\displaystyle {\phi }_i}$:

${\displaystyle X=\sum _{i=1}^N{Y}_i{\phi }_i=\sum _{i=1}^N⟨{\phi }_i,X⟩{\phi }_i}$

As in the continuous case, we may reduce the dimensionality of the problem by truncating the sum at some${\displaystyle K\in \{1,\dots ,N\}}$ such that

${\displaystyle \frac{\sum _{i=1}^K{\lambda }_i}{\sum _{i=1}^N{\lambda }_i}\ge \alpha }$

where α is the explained variance threshold we wish to set.

We can also reduce the dimensionality through the use of multilevel dominant eigenvector estimation (MDEE).^[6]

There are numerous equivalent characterizations of theWiener process which is a mathematical formalization ofBrownian motion. Here we regard it as the centered standard Gaussian processW_t with covariance function

${\displaystyle K_W(t,s)=\mathrm{cov}(W_t,W_s)=min(s,t).}$

We restrict the time domain to [a,b]=[0,1] without loss of generality.

The eigenvectors of the covariance kernel are easily determined. These are

${\displaystyle e_k(t)=\sqrt{2}\sin \left(\left(k-\frac{1}{2}\right)\pi t\right)}$

and the corresponding eigenvalues are

${\displaystyle {\lambda }_k=\frac{1}{(k-\frac{1}{2}{)}^2{\pi }^2}.}$

This gives the following representation of the Wiener process:

Theorem. There is a sequence {Z_i}_i of independent Gaussian random variables with mean zero and variance 1 such that

${\displaystyle W_t=\sqrt{2}\sum _{k=1}^{\mathrm{\infty }}{Z}_k\frac{\sin \left(\left(k-\frac{1}{2}\right)\pi t\right)}{\left(k-\frac{1}{2}\right)\pi }.}$

Note that this representation is only valid for${\displaystyle t\in [0,1].}$ On larger intervals, the increments are not independent. As stated in the theorem, convergence is in the L² norm and uniform in t.

Similarly theBrownian bridge${\displaystyle B_t=W_t-t{W}_1}$ which is astochastic process with covariance function

${\displaystyle K_B(t,s)=min(t,s)-ts}$

can be represented as the series

${\displaystyle B_t=\sum _{k=1}^{\mathrm{\infty }}{Z}_k\frac{\sqrt{2}\sin (k\pi t)}{k\pi }}$

This sectionneeds expansion. You can help byadding missing information.(July 2010)

Adaptive optics systems sometimes use K–L functions to reconstruct wave-front phase information (Dai 1996, JOSA A).Karhunen–Loève expansion is closely related to theSingular Value Decomposition. The latter has myriad applications in image processing, radar, seismology, and the like. If one has independent vector observations from a vector valued stochastic process then the left singular vectors aremaximum likelihood estimates of the ensemble KL expansion.

In communication, we usually have to decide whether a signal from a noisy channel contains valuable information. The following hypothesis testing is used for detecting continuous signals(t) from channel outputX(t),N(t) is the channel noise, which is usually assumed zero mean Gaussian process with correlation function${\displaystyle R_N(t,s)=E[N(t)N(s)]}$

${\displaystyle H:X(t)=N(t),}$

${\displaystyle K:X(t)=N(t)+s(t),t\in (0,T)}$

When the channel noise is white, its correlation function is

${\displaystyle R_N(t)=\frac{1}{2}{N}_0\delta (t),}$

and it has constant power spectrum density. In physically practical channel, the noise power is finite, so:

Then the noise correlation function is sinc function with zeros at${\displaystyle \frac{n}{2\omega },n\in \mathbf{Z}.}$ Since are uncorrelated and gaussian, they are independent. Thus we can take samples fromX(t) with time spacing

${\displaystyle \mathrm{\Delta }t=\frac{n}{2\omega }\text{ within }(0{,}^{″}{T}^{″}).}$

Let${\displaystyle X_i=X(i\mathrm{\Delta }t)}$. We have a total of${\displaystyle n=\frac{T}{\mathrm{\Delta }t}=T(2\omega )=2\omega T}$ i.i.d observations${\displaystyle \{X_1,X_2,\dots ,X_n\}}$ to develop the likelihood-ratio test. Define signal${\displaystyle S_i=S(i\mathrm{\Delta }t)}$, the problem becomes,

${\displaystyle H:X_i=N_i,}$

${\displaystyle K:X_i=N_i+S_i,i=1,2,\dots ,n.}$

The log-likelihood ratio

${\displaystyle \mathcal{L}(\underset{\_}{x})=\log \frac{\sum _{i=1}^n(2S_i{x}_i-S_i^2)}{2{\sigma }^2}\iff \mathrm{\Delta }t\sum _{i=1}^n{S}_i{x}_i=\sum _{i=1}^{n}S(i\mathrm{\Delta }t)x(i\mathrm{\Delta }t)\mathrm{\Delta }t\gtrless {\lambda }_{\cdot }2}$

Ast → 0, let:

${\displaystyle G={\int }_0^{T}S(t)x(t)dt.}$

ThenG is the test statistics and theNeyman–Pearson optimum detector is

AsG is Gaussian, we can characterize it by finding its mean and variances. Then we get

${\displaystyle H:G\sim N\left(0,\frac{1}{2}{N}_{0}E\right)}$

${\displaystyle K:G\sim N\left(E,\frac{1}{2}{N}_{0}E\right)}$

where

${\displaystyle \mathbf{E}={\int }_0^T{S}^2(t)dt}$

is the signal energy.

The false alarm error

${\displaystyle \alpha ={\int }_{G_0}^{\mathrm{\infty }}N\left(0,\frac{1}{2}{N}_{0}E\right)dG\Rightarrow G_0=\sqrt{\frac{1}{2}{N}_{0}E}{\mathrm{\Phi }}^{-1}(1-\alpha )}$

And the probability of detection:

${\displaystyle \beta ={\int }_{G_0}^{\mathrm{\infty }}N\left(E,\frac{1}{2}{N}_{0}E\right)dG=1-\mathrm{\Phi }\left(\frac{G_0-E}{\sqrt{\frac{1}{2}{N}_{0}E}}\right)=\mathrm{\Phi }\left(\sqrt{\frac{2E}{N_0}}-{\mathrm{\Phi }}^{-1}(1-\alpha )\right),}$

where Φ is the cdf of standard normal, or Gaussian, variable.

When N(t) is colored (correlated in time) Gaussian noise with zero mean and covariance function${\displaystyle R_N(t,s)=E[N(t)N(s)],}$ we cannot sample independent discrete observations by evenly spacing the time. Instead, we can use K–L expansion to decorrelate the noise process and get independent Gaussian observation 'samples'. The K–L expansion ofN(t):

where${\displaystyle N_i=\int N(t){\mathrm{\Phi }}_i(t)dt}$ and the orthonormal bases${\displaystyle \{{\mathrm{\Phi }}_{i}t\}}$ are generated by kernel${\displaystyle R_N(t,s)}$, i.e., solution to

${\displaystyle {\int }_0^T{R}_N(t,s){\mathrm{\Phi }}_i(s)ds={\lambda }_i{\mathrm{\Phi }}_i(t),\mathrm{var}[N_i]={\lambda }_i.}$