Instatistics, thejackknife (jackknife cross-validation) is across-validation technique and, therefore, a form ofresampling.It is especially useful forbias andvariance estimation. The jackknife pre-dates other common resampling methods such as thebootstrap. Given a sample of size${\displaystyle n}$, a jackknifeestimator can be built by aggregating the parameter estimates from each subsample of size${\displaystyle (n-1)}$ obtained by omitting one observation.^[1] The jackknife is a linear approximation of thebootstrap.^[2]

The jackknife technique was developed byMaurice Quenouille (1924–1973) from 1949 and refined in 1956.John Tukey expanded on the technique in 1958 and proposed the name "jackknife" because, like a physicaljack-knife (a compact folding knife), it is arough-and-ready tool that can improvise a solution for a variety of problems even though specific problems may be more efficiently solved with a purpose-designed tool.^[2]

The jackknifeestimator of a parameter is found by systematically leaving out each observation from a dataset and calculating the parameter estimate over the remaining observations and then aggregating these calculations.

For example, if the parameter to be estimated is the population mean of random variable${\displaystyle x}$, then for a given set ofi.i.d. observations${\displaystyle x_1,...,x_n}$ the natural estimator is the sample mean:

${\displaystyle \overline{x}=\frac{1}{n}\sum _{i=1}^n{x}_i=\frac{1}{n}\sum _{i\in [n]}{x}_i,}$

where the last sum used another way to indicate that the index${\displaystyle i}$ runs over the set${\displaystyle [n]=\{1,\dots ,n\}}$.

Then we proceed as follows: For each${\displaystyle i\in [n]}$ we compute the mean${\displaystyle {\overline{x}}_{(i)}}$ of the jackknife subsample consisting of all but the${\displaystyle i}$-th data point, and this is called the${\displaystyle i}$-th jackknife replicate:

${\displaystyle {\overline{x}}_{(i)}=\frac{1}{n-1}\sum _{j\in [n],j\ne i}{x}_j,i=1,\dots ,n.}$

It could help to think that these${\displaystyle n}$ jackknife replicates${\displaystyle {\overline{x}}_{(1)},\dots ,{\overline{x}}_{(n)}}$ approximate the distribution of the sample mean${\displaystyle \overline{x}}$. A larger${\displaystyle n}$ improves the approximation. Then finally to get the jackknife estimator, the${\displaystyle n}$ jackknife replicates are averaged:

${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=\frac{1}{n}\sum _{i=1}^n{\overline{x}}_{(i)}.}$

One may ask about the bias and the variance of${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$. From the definition of${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$ as the average of the jackknife replicates one could try to calculate explicitly. The bias is a trivial calculation, but the variance of${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$ is more involved since the jackknife replicates are not independent.

For the special case of the mean, one can show explicitly that the jackknife estimate equals the usual estimate:

${\displaystyle \frac{1}{n}\sum _{i=1}^n{\overline{x}}_{(i)}=\overline{x}.}$

This establishes the identity${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=\overline{x}}$. Then taking expectations we get${\displaystyle E[{\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}]=E[\overline{x}]=E[x]}$, so${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$ is unbiased, while taking variance we get${\displaystyle V[{\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}]=V[\overline{x}]=V[x]/n}$. However, these properties do not generally hold for parameters other than the mean.

This simple example for the case of mean estimation is just to illustrate the construction of a jackknife estimator, while the real subtleties (and the usefulness) emerge for the case of estimating other parameters, such as higher moments than the mean or other functionals of the distribution.

${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$ could be used to construct an empirical estimate of the bias of${\displaystyle \overline{x}}$, namely${\displaystyle \widehat{\mathrm{bias}}(\overline{x}{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=c({\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}-\overline{x})}$ with some suitable factor${\displaystyle c>0}$, although in this case we know that${\displaystyle {\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=\overline{x}}$ so this construction does not add any meaningful knowledge, but it gives the correct estimation of the bias (which is zero).

A jackknife estimate of the variance of${\displaystyle \overline{x}}$ can be calculated from the variance of the jackknife replicates${\displaystyle {\overline{x}}_{(i)}}$:^[3][4]

${\displaystyle \widehat{\mathrm{var}}(\overline{x}{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=\frac{n-1}{n}\sum _{i=1}^n({\overline{x}}_{(i)}-{\overline{x}}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}{)}^2=\frac{1}{n(n-1)}\sum _{i=1}^n(x_i-\overline{x}{)}^2.}$

The left equality defines the estimator${\displaystyle \widehat{\mathrm{var}}(\overline{x}{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}}$ and the right equality is an identity that can be verified directly. Then taking expectations we get${\displaystyle E[\widehat{\mathrm{var}}(\overline{x}{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}]=V[x]/n=V[\overline{x}]}$, so this is an unbiased estimator of the variance of${\displaystyle \overline{x}}$.

The jackknife technique can be used to estimate (and correct) the bias of an estimator calculated over the entire sample.

Suppose${\displaystyle \theta }$ is the target parameter of interest, which is assumed to be some functional of the distribution of${\displaystyle x}$. Based on a finite set of observations${\displaystyle x_1,...,x_n}$, which is assumed to consist ofi.i.d. copies of${\displaystyle x}$, the estimator${\displaystyle \hat{\theta }}$ is constructed:

${\displaystyle \hat{\theta }=f_n(x_1,\dots ,x_n).}$

The value of${\displaystyle \hat{\theta }}$ is sample-dependent, so this value will change from one random sample to another.

By definition, the bias of${\displaystyle \hat{\theta }}$ is as follows:

${\displaystyle \text{bias}(\hat{\theta })=E[\hat{\theta }]-\theta .}$

One may wish to compute several values of${\displaystyle \hat{\theta }}$ from several samples, and average them, to calculate an empirical approximation of${\displaystyle E[\hat{\theta }]}$, but this is impossible when there are no "other samples" when the entire set of available observations${\displaystyle x_1,...,x_n}$ was used to calculate${\displaystyle \hat{\theta }}$. In this kind of situation the jackknife resampling technique may be of help.

We construct the jackknife replicates:

${\displaystyle {\hat{\theta }}_{(1)}=f_{n-1}(x_2,x_3\dots ,x_n)}$

${\displaystyle {\hat{\theta }}_{(2)}=f_{n-1}(x_1,x_3,\dots ,x_n)}$

${\displaystyle ⋮}$

${\displaystyle {\hat{\theta }}_{(n)}=f_{n-1}(x_1,x_2,\dots ,x_{n-1})}$

where each replicate is a "leave-one-out" estimate based on the jackknife subsample consisting of all but one of the data points:

${\displaystyle {\hat{\theta }}_{(i)}=f_{n-1}(x_1,\dots ,x_{i-1},x_{i+1},\dots ,x_n)i=1,\dots ,n.}$

Then we define their average:

${\displaystyle {\hat{\theta }}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=\frac{1}{n}\sum _{i=1}^n{\hat{\theta }}_{(i)}}$

The jackknife estimate of the bias of${\displaystyle \hat{\theta }}$ is given by:

${\displaystyle \widehat{\text{bias}}(\hat{\theta }{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=(n-1)({\hat{\theta }}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}-\hat{\theta })}$

and the resulting bias-corrected jackknife estimate of${\displaystyle \theta }$ is given by:

${\displaystyle {\hat{\theta }}_{\text{jack}}^{\ast }=\hat{\theta }-\widehat{\text{bias}}(\hat{\theta }{)}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}=n\hat{\theta }-(n-1){\hat{\theta }}_{\mathrm{j}\mathrm{a}\mathrm{c}\mathrm{k}}.}$

This removes the bias in the special case that the bias is${\displaystyle O(n^{-1})}$ and reduces it to${\displaystyle O(n^{-2})}$ in other cases.^[2]

The jackknife technique can be also used to estimate the variance of an estimator calculated over the entire sample.

• Berger, Y.G. (2007). "A jackknife variance estimator for unistage stratified samples with unequal probabilities".Biometrika.94 (4):953–964.doi:10.1093/biomet/asm072.
• Berger, Y.G.; Rao, J.N.K. (2006)."Adjusted jackknife for imputation under unequal probability sampling without replacement".Journal of the Royal Statistical Society, Series B.68 (3):531–547.doi:10.1111/j.1467-9868.2006.00555.x.
• Berger, Y.G.; Skinner, C.J. (2005). "A jackknife variance estimator for unequal probability sampling".Journal of the Royal Statistical Society, Series B.67 (1):79–89.doi:10.1111/j.1467-9868.2005.00489.x.
• Jiang, J.; Lahiri, P.; Wan, S-M. (2002)."A unified jackknife theory for empirical best prediction with M-estimation".The Annals of Statistics.30 (6):1782–810.doi:10.1214/aos/1043351257.
• Jones, H.L. (1974). "Jackknife estimation of functions of stratum means".Biometrika.61 (2):343–348.doi:10.2307/2334363.JSTOR 2334363.
• Kish, L.; Frankel, M.R. (1974). "Inference from complex samples".Journal of the Royal Statistical Society, Series B.36 (1):1–37.
• Krewski, D.; Rao, J.N.K. (1981)."Inference from stratified samples: properties of the linearization, jackknife and balanced repeated replication methods".The Annals of Statistics.9 (5):1010–1019.doi:10.1214/aos/1176345580.
• Quenouille, M.H. (1956). "Notes on bias in estimation".Biometrika.43 (3–4):353–360.doi:10.1093/biomet/43.3-4.353.
• Rao, J.N.K.; Shao, J. (1992). "Jackknife variance estimation with survey data under hot deck imputation".Biometrika.79 (4):811–822.doi:10.1093/biomet/79.4.811.
• Rao, J.N.K.; Wu, C.F.J.; Yue, K. (1992). "Some recent work on resampling methods for complex surveys".Survey Methodology.18 (2):209–217.
• Shao, J. and Tu, D. (1995). The Jackknife and Bootstrap. Springer-Verlag, Inc.
• Tukey, J.W. (1958). "Bias and confidence in not-quite large samples (abstract)".The Annals of Mathematical Statistics.29 (2): 614.
• Wu, C.F.J. (1986)."Jackknife, Bootstrap and other resampling methods in regression analysis".The Annals of Statistics.14 (4):1261–1295.doi:10.1214/aos/1176350142.

• Cameron, Adrian; Trivedi, Pravin K. (2005).Microeconometrics : methods and applications. Cambridge New York: Cambridge University Press.ISBN 9780521848053.
• Efron, Bradley;Stein, Charles (May 1981)."The Jackknife Estimate of Variance".The Annals of Statistics.9 (3):586–596.doi:10.1214/aos/1176345462.JSTOR 2240822.
• Efron, Bradley (1982).The jackknife, the bootstrap, and other resampling plans. Philadelphia, PA: Society for Industrial and Applied Mathematics.ISBN 9781611970319.
• Quenouille, Maurice H. (September 1949)."Problems in Plane Sampling".The Annals of Mathematical Statistics.20 (3):355–375.doi:10.1214/aoms/1177729989.JSTOR 2236533.
• Quenouille, Maurice H. (1956). "Notes on Bias in Estimation".Biometrika.43 (3–4):353–360.doi:10.1093/biomet/43.3-4.353.JSTOR 2332914.
• Tukey, John W. (1958)."Bias and confidence in not quite large samples (abstract)".The Annals of Mathematical Statistics.29 (2): 614.doi:10.1214/aoms/1177706647.

• Computational statistics
• Resampling (statistics)

• Articles with short description
• Short description matches Wikidata