Basic Statistics

Basic descriptive statistics operations.

Given a list with elements, themeanMean[list] is defined to be.

ThevarianceVariance[list] is defined to be, for real data. (For complex data.)

Thestandard deviationStandardDeviation[list] is defined to be.

If the elements inlist are thought of as being selected at random according to some probability distribution, then the mean gives an estimate of where the center of the distribution is located, while the standard deviation gives an estimate of how wide the dispersion in the distribution is.

ThemedianMedian[list] effectively gives the value at the halfway point in the sorted version oflist. It is often considered a more robust measure of the center of a distribution than the mean, since it depends less on outlying values.

The^thquantileQuantile[list,q] effectively gives the value that is of the way through the sorted version oflist.

For a list of length, the Wolfram Language definesQuantile[list,q] to bes[[Ceiling[nq]]], where isSort[list,Less].

There are, however, about 10 other definitions of quantile in use, all potentially giving slightly different results. The Wolfram Language covers the common cases by introducing fourquantile parameters in the formQuantile[list,q,{{a,b},{c,d}}]. The parameters and in effect define where in the list should be considered a fraction of the way through. If this corresponds to an integer position, then the element at that position is taken to be the^th quantile. If it is not an integer position, then a linear combination of the elements on either side is used, as specified by and.

The position in a sorted list for the^th quantile is taken to be. If is an integer, then the quantile is. Otherwise, it is, with the indices taken to be or if they are out of range.

Common choices for quantile parameters.

Whenever, the value of the^th quantile is always equal to some actual element inlist, so that the result changes discontinuously as varies. For, the^th quantile interpolates linearly between successive elements inlist.Median is defined to use such an interpolation.

Note thatQuantile[list,q] yieldsquartiles when andpercentiles when.

Handling multidimensional data.

Sometimes each item in your data may involve a list of values. The basic statistics functions in the Wolfram Language automatically apply to all corresponding elements in these lists.

This separately finds the mean of each "column" of data:

Note that you can extract the elements in the^th "column" of a multidimensional list usinglist[[All,i]].

Descriptive Statistics

Descriptive statistics refers to properties of distributions, such as location, dispersion, and shape. The functions described here compute descriptive statistics of lists of data. You can calculate some of the standard descriptive statistics for various known distributions by using the functions described in"Continuous Distributions" and"Discrete Distributions".

The statistics are calculated assuming that each value of data has probability equal to, where is the number of elements in the data.

Location statistics.

Location statistics describe where the data is located. The most common functions include measures of central tendency like the mean, median, and mode.Quantile[data,q] gives the location before which percent of the data lie. In other words,Quantile gives a value such that the probability that is less than or equal to and the probability that is greater than or equal to.

Here is a dataset:

This finds the mean and median of the data:

This is the mean when the smallest entry in the list is excluded.TrimmedMean allows you to describe the data with removed outliers:

Dispersion statistics.

Dispersion statistics summarize the scatter or spread of the data. Most of these functions describe deviation from a particular location. For instance, variance is a measure of deviation from the mean, and standard deviation is just the square root of the variance.

This gives an unbiased estimate for the variance of the data with as the divisor:

This compares three types of deviation:

Covariance and correlation statistics.

Covariance is the multivariate extension of variance. For two vectors of equal length, the covariance is a number. For a single matrixm, thei,j^th element of the covariance matrix is the covariance between thei^th andj^th columns ofm. For two matricesm₁ andm₂, thei,j^th element of the covariance matrix is the covariance between thei^th column ofm₁ and thej^th column ofm₂.

While covariance measures dispersion, correlation measures association. The correlation between two vectors is equivalent to the covariance between the vectors divided by the standard deviations of the vectors. Likewise, the elements of a correlation matrix are equivalent to the elements of the corresponding covariance matrix scaled by the appropriate column standard deviations.

This gives the covariance betweendata and a random vector:

Here is a random matrix:

This is the correlation matrix for the matrixm:

This is the covariance matrix:

Scaling the covariance matrix terms by the appropriate standard deviations gives the correlation matrix:

Shape statistics.

You can get some information about the shape of a distribution using shape statistics. Skewness describes the amount of asymmetry. Kurtosis measures the concentration of data around the peak and in the tails versus the concentration in the flanks.

Skewness is calculated by dividing the third central moment by the cube of the population standard deviation.Kurtosis is calculated by dividing the fourth central moment by the square of the population variance of the data, equivalent toCentralMoment[data,2]. (The population variance is the second central moment, and the population standard deviation is its square root.)

QuartileSkewness is calculated from the quartiles of data. It is equivalent to, where,, and are the first, second, and third quartiles respectively.

Here is the second central moment of the data:

A negative value for skewness indicates that the distribution underlying the data has a long left‐sided tail:

Expected values.

The expectation or expected value of a function is for the list of values,,…,. Many descriptive statistics are expectations. For instance, the mean is the expected value of, and the^th central moment is the expected value of where is the mean of the.

Here is the expected value of theLog of the data:

Discrete Distributions

The functions described here are among the most commonly used discrete univariate statistical distributions. You can compute their densities, means, variances, and other related properties. The distributions themselves are represented in the symbolic formname[param₁,param₂,…]. Functions such asMean, which give properties of statistical distributions, take the symbolic representation of the distribution as an argument."Continuous Distributions" describes many continuous statistical distributions.

Discrete statistical distributions.

Most of the common discrete statistical distributions can be understood by considering a sequence of trials, each with two possible outcomes, for example, success and failure.

TheBernoulli distributionBernoulliDistribution[p] is the probability distribution for a single trial in which success, corresponding to value 1, occurs with probabilityp, and failure, corresponding to value 0, occurs with probability1-p.

Thebinomial distributionBinomialDistribution[n,p] is the distribution of the number of successes that occur inn independent trials, where the probability of success in each trial isp.

Thenegative binomial distributionNegativeBinomialDistribution[n,p] for positive integern is the distribution of the number of failures that occur in a sequence of trials beforen successes have occurred, where the probability of success in each trial isp. The distribution is defined for any positiven, though the interpretation ofn as the number of successes andp as the success probability no longer holds ifn is not an integer.

Thebeta binomial distributionBetaBinomialDistribution[α,β,n] is a mixture of binomial and beta distributions. ABetaBinomialDistribution[α,β,n] random variable follows aBinomialDistribution[n,p] distribution, where the success probabilityp is itself a random variable following the beta distributionBetaDistribution[α,β]. Thebeta negative binomial distributionBetaNegativeBinomialDistribution[α,β,n] is a similar mixture of the beta and negative binomial distributions.

Thegeometric distributionGeometricDistribution[p] is the distribution of the total number of trials before the first success occurs, where the probability of success in each trial isp.

Thehypergeometric distributionHypergeometricDistribution[n,n_succ,n_tot] is used in place of the binomial distribution for experiments in which then trials correspond to sampling without replacement from a population of sizen_tot withn_succ potential successes.

Thediscrete uniform distributionDiscreteUniformDistribution[{i_min,i_max}] represents an experiment with multiple equally probable outcomes represented by integersi_min throughi_max.

ThePoisson distributionPoissonDistribution[μ] describes the number of events that occur in a given time period whereμ is the average number of events per period.

The terms in the series expansion of about are proportional to the probabilities of a discrete random variable following thelogarithmic series distributionLogSeriesDistribution[θ]. The distribution of the number of items of a product purchased by a buyer in a specified interval is sometimes modeled by this distribution.

TheZipf distributionZipfDistribution[ρ], sometimes referred to as the zeta distribution, was first used in linguistics and its use has been extended to model rare events.

Some functions of statistical distributions.

Distributions are represented in symbolic form.PDF[dist,x] evaluates the mass function atx ifx is a numerical value, and otherwise leaves the function in symbolic form whenever possible. Similarly,CDF[dist,x] gives the cumulative distribution andMean[dist] gives the mean of the specified distribution. The table above gives a sampling of some of the more common functions available for distributions. For a more complete description of these functions, see the description of their continuous analogues in"Continuous Distributions".

Here is a symbolic representation of the binomial distribution for 34 trials, each having probability 0.3 of success:

This is the mean of the distribution:

You can get the expression for the mean by using symbolic variables as arguments:

Here is the 50% quantile, which is equal to the median:

This gives the expected value of with respect to the binomial distribution:

The elements of this matrix are pseudorandom numbers from the binomial distribution:

Continuous Distributions

The functions described here are among the most commonly used continuous univariate statistical distributions. You can compute their densities, means, variances, and other related properties. The distributions themselves are represented in the symbolic formname[param₁,param₂,…]. Functions such asMean, which give properties of statistical distributions, take the symbolic representation of the distribution as an argument."Discrete Distributions" describes many common discrete univariate statistical distributions.

Distributions related to the normal distribution.

Thelognormal distributionLogNormalDistribution[μ,σ] is the distribution followed by the exponential of a normally distributed random variable. This distribution arises when many independent random variables are combined in a multiplicative fashion. Thehalf-normal distributionHalfNormalDistribution[θ] is proportional to the distributionNormalDistribution[0,1/(θSqrt[2/π])] limited to the domain.

Theinverse Gaussian distributionInverseGaussianDistribution[μ,λ], sometimes called the Wald distribution, is the distribution of first passage times in Brownian motion with positive drift.

Distributions related to normally distributed samples.

If,…, are independent normal random variables with unit variance and mean zero, then has a distribution with degrees of freedom. If a normal variable is standardized by subtracting its mean and dividing by its standard deviation, then the sum of squares of such quantities follows this distribution. The distribution is most typically used when describing the variance of normal samples.

If follows a distribution with degrees of freedom, follows theinverse distributionInverseChiSquareDistribution[ν]. Ascaled inverse distribution with degrees of freedom and scale can be given asInverseChiSquareDistribution[ν,ξ]. Inverse distributions are commonly used as prior distributions for the variance in Bayesian analysis of normally distributed samples.

A variable that has aStudent distribution can also be written as a function of normal random variables. Letand be independent random variables, where is a standard normal distribution and is a variable with degrees of freedom. In this case, has a distribution with degrees of freedom. The Student distribution is symmetric about the vertical axis, and characterizes the ratio of a normal variable to its standard deviation. Location and scale parameters can be included asμ andσ inStudentTDistribution[μ,σ,ν]. When, the distribution is the same as the Cauchy distribution.

The‐ratio distribution is the distribution of the ratio of two independent variables divided by their respective degrees of freedom. It is commonly used when comparing the variances of two populations in hypothesis testing.

Distributions that are derived from normal distributions with nonzero means are callednoncentral distributions.

The sum of the squares of normally distributed random variables with variance and nonzero means follows anoncentral distributionNoncentralChiSquareDistribution[ν,λ]. The noncentrality parameter is the sum of the squares of the means of the random variables in the sum. Note that in various places in the literature, or is used as the noncentrality parameter.

Thenoncentral Student distributionNoncentralStudentTDistribution[ν,δ] describes the ratio where is a central random variable with degrees of freedom, and is an independent normally distributed random variable with variance and mean.

Thenoncentral‐ratio distributionNoncentralFRatioDistribution[n,m,λ] is the distribution of the ratio of to, where is a noncentral random variable with noncentrality parameter and degrees of freedom and is a central random variable with degrees of freedom.

Piecewise linear distributions.

Thetriangular distributionTriangularDistribution[{a,b},c] is a triangular distribution for with maximum probability at and. If is,TriangularDistribution[{a,b},c] is the symmetric triangular distributionTriangularDistribution[{a,b}].

Theuniform distributionUniformDistribution[{min,max}], commonly referred to as the rectangular distribution, characterizes a random variable whose value is everywhere equally likely. An example of a uniformly distributed random variable is the location of a point chosen randomly on a line frommin tomax.

Other continuous statistical distributions.

If is uniformly distributed on[-π,π], then the random variable follows aCauchy distributionCauchyDistribution[a,b], with and.

When and, thegamma distributionGammaDistribution[α,λ] describes the distribution of a sum of squares of-unit normal random variables. This form of the gamma distribution is called a distribution with degrees of freedom. When, the gamma distribution takes on the form of theexponential distributionExponentialDistribution[λ], often used in describing the waiting time between events.

If a random variable follows thegamma distributionGammaDistribution[α,β], follows theinverse gamma distributionInverseGammaDistribution[α,1/β]. If a random variable followsInverseGammaDistribution[1/2,σ/2], follows aLévy distributionLevyDistribution[μ,σ].

When and have independent gamma distributions with equal scale parameters, the random variable follows thebeta distributionBetaDistribution[α,β], where and are the shape parameters of the gamma variables.

The distributionChiDistribution[ν] is followed by the square root of a random variable. For, the distribution is identical toHalfNormalDistribution[θ] with. For, the distribution is identical to theRayleigh distributionRayleighDistribution[σ] with. For, the distribution is identical to theMaxwell–Boltzmann distributionMaxwellDistribution[σ] with.

TheLaplace distributionLaplaceDistribution[μ,β] is the distribution of the difference of two independent random variables with identical exponential distributions. Thelogistic distributionLogisticDistribution[μ,β] is frequently used in place of the normal distribution when a distribution with longer tails is desired.

ThePareto distributionParetoDistribution[k,α] may be used to describe income, with representing the minimum income possible.

TheWeibull distributionWeibullDistribution[α,β] is commonly used in engineering to describe the lifetime of an object. Theextreme value distributionExtremeValueDistribution[α,β] is the limiting distribution for the largest values in large samples drawn from a variety of distributions, including the normal distribution. The limiting distribution for the smallest values in such samples is theGumbel distribution,GumbelDistribution[α,β]. The names "extreme value" and "Gumbel distribution" are sometimes used interchangeably because the distributions of the largest and smallest extreme values are related by a linear change of variable. The extreme value distribution is also sometimes referred to as the log‐Weibull distribution because of logarithmic relationships between an extreme value-distributed random variable and a properly shifted and scaled Weibull-distributed random variable.

Some functions of statistical distributions.

The preceding table gives a list of some of the more common functions available for distributions in the Wolfram Language.

Thecumulative distribution function (CDF) at is given by the integral of theprobability density function (PDF) up to. The PDF can therefore be obtained by differentiating the CDF (perhaps in a generalized sense). In this package the distributions are represented in symbolic form.PDF[dist,x] evaluates the density at if is a numerical value, and otherwise leaves the function in symbolic form. Similarly,CDF[dist,x] gives the cumulative distribution.

The inverse CDFInverseCDF[dist,q] gives the value of at whichCDF[dist,x] reaches. The median is given byInverseCDF[dist,1/2]. Quartiles, deciles, and percentiles are particular values of the inverse CDF. Quartile skewness is equivalent to, where,, and are the first, second, and third quartiles, respectively. Inverse CDFs are used in constructing confidence intervals for statistical parameters.InverseCDF[dist,q] andQuantile[dist,q] are equivalent for continuous distributions.

The meanMean[dist] is the expectation of the random variable distributed according todist and is usually denoted by. The mean is given by, where is the PDF of the distribution. The varianceVariance[dist] is given by. The square root of the variance is called the standard deviation, and is usually denoted by.

TheSkewness[dist] andKurtosis[dist] functions give shape statistics summarizing the asymmetry and the peakedness of a distribution, respectively. Skewness is given by and kurtosis is given by.

The characteristic functionCharacteristicFunction[dist,t] is given by. In the discrete case,. Each distribution has a unique characteristic function, which is sometimes used instead of the PDF to define a distribution.

The expected valueExpectation[g[x],xdist] of a functiong is given by. In the discrete case, the expected value ofg is given by.

RandomVariate[dist] gives pseudorandom numbers from the specified distribution.

This gives a symbolic representation of the gamma distribution with and:

Here is the cumulative distribution function evaluated at10:

This is the cumulative distribution function. It is given in terms of the built‐in functionGammaRegularized:

Here is a plot of the cumulative distribution function:

This is a pseudorandom array with elements distributed according to the gamma distribution:

Partitioning Data into Clusters

Cluster analysis is an unsupervised learning technique used for classification of data. Data elements are partitioned into groups called clusters that represent proximate collections of data elements based on a distance or dissimilarity function. Identical element pairs have zero distance or dissimilarity, and all others have positive distance or dissimilarity.

General clustering function.

The data argument ofFindClusters can be a list of data elements, associations, or rules indexing elements and labels.

Ways of specifying data inFindClusters.

FindClusters works for a variety of data types, including numerical, textual, and image, as well as Boolean vectors, dates and times. All data elementse_i must have the same dimensions.

Here is a list of numbers:

FindClusters clusters the numbers based on their proximity:

The rule-based data syntax allows for clustering data elements and returning labels for those elements.

Here two-dimensional points are clustered and labeled with their positions in the data list:

The rule-based data syntax can also be used to cluster data based on parts of each data entry. For instance, you might want to cluster data in a data table while ignoring particular columns in the table.

Here is a list of data entries:

This clusters the data while ignoring the first two elements in each data entry:

In principle, it is possible to cluster points given in an arbitrary number of dimensions. However, it is difficult at best to visualize the clusters above two or three dimensions. To compare optional methods in this documentation, an easily visualizable set of two-dimensional data will be used.

The following commands define a set of 300 two-dimensional data points chosen to group into four somewhat nebulous clusters:

This clusters the data based on the proximity of points:

Here is a plot of the clusters:

With the default settings,FindClusters has found the four clusters of points.

You can also directFindClusters to find a specific number of clusters.

This shows the effect of choosing 3 clusters:

This shows the effect of choosing 5 clusters:

Options forFindClusters.

In principle, clustering techniques can be applied to any set of data. All that is needed is a measure of how far apart each element in the set is from other elements, that is, a function giving the distance between elements.

FindClusters[{e₁,e₂,…},DistanceFunction->f] treats pairs of elements as being less similar when their distancesf[e_i,e_j] are larger. The functionf can be any appropriate distance or dissimilarity function. A dissimilarity function f satisfies the following:

If thee_i are vectors of numbers,FindClusters by default uses a squared Euclidean distance. If thee_i are lists of BooleanTrue andFalse (or 0 and 1) elements,FindClusters by default uses a dissimilarity based on the normalized fraction of elements that disagree. If thee_i are strings,FindClusters by default uses a distance function based on the number of point changes needed to get from one string to another.

Distance functions for numerical data.

This shows the clusters indatapairs found using a Manhattan distance:

Dissimilarities for Boolean vectors are typically calculated by comparing the elements of two Boolean vectors and pairwise. It is convenient to summarize each dissimilarity function in terms of, where is the number of corresponding pairs of elements in and, respectively, equal to and. The number counts the pairs in, with and being either 0 or 1. If the Boolean values areTrue andFalse,True is equivalent to 1 andFalse is equivalent to 0.

Dissimilarity functions for Boolean data.

Here is some Boolean data:

These are the clusters found using the default dissimilarity for Boolean data:

Dissimilarity functions for string data.

The edit distance is determined by counting the number of deletions, insertions, and substitutions required to transform one string into another while preserving the ordering of characters. In contrast, the Damerau–Levenshtein distance counts the number of deletions, insertions, substitutions, and transpositions, while the Hamming distance counts only the number of substitutions.

Here is some string data:

This clusters the string data using the edit distance:

TheMethod option can be used to specify different methods of clustering.

Explicit settings for theMethod option.

By default,FindClusters tries different methods and selects the best clustering.

The methods"KMeans" and"KMedoids" determine how to cluster the data for a particular number of clustersk.

The methods"DBSCAN","JarvisPatrick","MeanShift","SpanningTree","NeighborhoodContraction", and"GaussianMixture" determine how to cluster the data without assuming any particular number of clusters.

The methods"Agglomerate","Spectral" and"SpanningTree" can be used in both cases.

This shows the clusters indatapairs found using the"KMeans" method:

This shows the clusters indatapairs found using the"GaussianMixture" method:

AdditionalMethod suboptions are available to allow for more control over the clustering. Available suboptions depend on theMethod chosen.

Suboption for all methods.

The suboption"NeighborhoodRadius" can be used in methods"DBSCAN","MeanShift","JarvisPatrick","NeighborhoodContraction", and"Spectral".

The suboptions"NeighborsNumber" and"SharedNeighborsNumber" can be used in methods"DBSCAN" and"JarvisPatrick", respectively.

The suboption"MaxEdgeLength" can be used in the method"SpanningTree".

The suboption"InitialCentroids" can be used in methods"KMeans" and"KMedoids".

The suboptionClusterDissimilarityFunction can be used in the method"Agglomerate".

The"NeighborhoodRadius" suboption can be used to control the average radius of the neighborhood of a generic point.

This shows different clusterings ofdatapairs found using the"NeighborhoodContraction" method by varying the"NeighborhoodRadius":

The"NeighborsNumber" suboption can be used to control the number of neighbors in the neighborhood of a generic point.

This shows different clusterings ofdatapairs found using the"DBSCAN" method by varying the"NeighborsNumber":

The"InitialCentroids" suboption can be used to change the initial configuration in the"KMeans" and"KMedoids" methods. Bad initial configurations may result in bad clusterings.

This shows different clusterings ofdatapairs found using the"KMeans" method by varying the"InitialCentroids":

WithMethod->{"Agglomerate",ClusterDissimilarityFunction->f}, the specified linkage functionf is used for agglomerative clustering.

Possible values for theClusterDissimilarityFunction suboption.

Linkage methods determine this intercluster dissimilarity, or fusion level, given the dissimilarities between member elements.

WithClusterDissimilarityFunction->f,f is a pure function that defines the linkage algorithm. Distances or dissimilarities between clusters are determined recursively using information about the distances or dissimilarities between unmerged clusters to determine the distances or dissimilarities for the newly merged cluster. The functionf defines a distance from a clusterk to the new cluster formed by fusing clustersi andj. The arguments supplied tof ared_ik,d_jk,d_ij,n_i,n_j, andn_k, whered is the distance between clusters andn is the number of elements in a cluster.

This shows different clusterings ofdatapairs found using the"Agglomerate" method by varying theClusterDissimilarityFunction:

TheCriterionFunction option can be used to select both the method to use and the best number of clusters.

This shows the result of clustering using different settings forCriterionFunction:

These are the clusters found using the defaultCriterionFunction with automatically selected number of clusters:

These are the clusters found using the"CalinskiHarabasz" index:

Using Nearest

Nearest is used to find elements in a list that are closest to a given data point.

Nearest function.

Nearest works with numeric lists, tensors, or a list of strings.

This finds the elements nearest to 4.5:

This finds 3 elements nearest to 4.5:

This finds all elements nearest to 4.5 within a radius of 2:

This finds the points nearest to{1,2} in 2D:

This finds the nearest string to "cat":

The rule-based data syntax lets you use nearest elements to return their labels.

Here two-dimensional points are labeled:

This labels the elements using successive integers:

IfNearest is to be applied repeatedly to the same numerical data, you can get significant performance gains by first generating aNearestFunction.

This generates a set of 10,000 points in 2D and aNearestFunction:

This finds points in the set that are closest to the 10 target points:

It takes much longer ifNearestFunction is not used:

Option forNearest.

For numerical data, by defaultNearest uses theEuclideanDistance. For strings,EditDistance is used.

Manipulating Numerical Data

When you have numerical data, it is often convenient to find a simple formula that approximates it. For example, you can try to "fit" a line or curve through the points in your data.

Fitting curves to linear combinations of functions.

This generates a table of the numerical values of the exponential function.Table is discussed in"Making Tables of Values":

This finds a least‐squares fit todata of the form. The elements ofdata are assumed to correspond to values,,… of:

This finds a fit of the form:

This gives a table of, pairs:

This finds a fit to the new data, of the form:

Fitting data to general forms.

This finds the best parameters for a linear fit:

This does a nonlinear fit:

One common way of picking out "signals" in numerical data is to find theFourier transform, or frequency spectrum, of the data.

Fourier transforms.

Here is a simple square pulse:

This takes the Fourier transform of the pulse:

Note that theFourier function in the Wolfram Language is defined with the sign convention typically used in the physical sciences—opposite to the one often used in electrical engineering."Discrete Fourier Transforms" gives more details.

Curve Fitting

There are many situations where one wants to find a formula that best fits a given set of data. One way to do this in the Wolfram Language is to useFit.

Basic linear fitting.

Here is a table of the first 20 primes:

Here is a plot of this "data":

This gives a linear fit to the list of primes. The result is the best linear combination of the functions1 andx:

Here is a plot of the fit:

Here is the fit superimposed on the original data:

This gives a quadratic fit to the data:

Here is a plot of the quadratic fit:

This shows the fit superimposed on the original data. The quadratic fit is better than the linear one:

Ways of specifying data.

If you give data in the form thenFit will assume that the successive correspond to values of a function at successive integer points. But you can also giveFit data that corresponds to the values of a function at arbitrary points, in one or more dimensions.

Multivariate fitting.

This gives a table of the values of,, and. You need to useFlatten to get it in the right form forFit:

This produces a fit to a function of two variables:

Fit takes a list of functions, and uses a definite and efficient procedure to find what linear combination of these functions gives the best least‐squares fit to your data. Sometimes, however, you may want to find anonlinear fit that does not just consist of a linear combination of specified functions. You can do this usingFindFit, which takes a function of any form, and then searches for values of parameters that yield the best fit to your data.

Searching for general fits to data.

This fits the list of primes to a simple linear combination of terms:

The result is the same as fromFit:

This fits to a nonlinear form, which cannot be handled byFit:

By default, bothFit andFindFit produceleast‐squares fits, which are defined to minimize the quantity, where the are residuals giving the difference between each original data point and its fitted value. One can, however, also consider fits based on other norms. If you set the optionNormFunction->u, thenFindFit will attempt to find the fit that minimizes the quantityu[r], wherer is the list of residuals. The default isNormFunction->Norm, corresponding to a least‐squares fit.

This uses the‐norm, which minimizes the maximum distance between the fit and the data. The result is slightly different from least‐squares:

FindFit works by searching for values of parameters that yield the best fit. Sometimes you may have to tell it where to start in doing this search. You can do this by giving parameters in the form.FindFit also has various options that you can set to control how it does its search.

Searching for general fits to data.

This gives a best fit subject to constraints on the parameters:

Options forFindFit.

Statistical Model Analysis

When fitting models to data, it is often useful to analyze how well the model fits the data and how well the fitting meets the assumptions of the model. For a number of common statistical models, this is accomplished in the Wolfram System by way of fitting functions that constructFittedModel objects.

Object for fitted model information.

FittedModel objects can be evaluated at a point or queried for results and diagnostic information. Diagnostics vary somewhat across model types. Available model fitting functions fit linear, generalized linear, and nonlinear models.

Functions that generateFittedModel objects.

This fits a linear model assuming values are 1, 2,…:

Here is the functional form of the fitted model:

This evaluates the model at:

Here is a shortened list of available results for the linear fitted model:

The major difference between model fitting functions such asLinearModelFit and functions such asFit andFindFit is the ability to easily obtain diagnostic information from theFittedModel objects. The results are accessible without refitting the model.

This gives the residuals for the fitting:

Here multiple results are obtained at once in a list:

Fitting options relevant to property computations can be passed toFittedModel objects to override defaults.

This gives default 95% confidence intervals:

Here 90% intervals are obtained:

Typical data for these model-fitting functions takes the same form as data in other fitting functions such asFit andFindFit.

Data specifications.

Linear models with assumed independent normally distributed errors are among the most common models for data. Models of this type can be fitted using theLinearModelFit function.

Linear model fitting.

Linear models have the form, where is the fitted or predicted value, the are parameters to be fitted, and the are functions of the predictor variables. The models are linear in the parameters. The can be any functions of the predictor variables. Quite often the are simply the predictor variables.

This fits a linear model to the first 20 primes:

Options for model specification and for model analysis are available.

Options forLinearModelFit.

TheWeights option specifies weight values for weighted linear regression. TheNominalVariables option specifies which predictor variables should be treated as nominal or categorical. WithNominalVariables->All, the model is an analysis of variance (ANOVA) model. WithNominalVariables->{x₁,…,x_i-1,x_i+1,…,x_n} the model is an analysis of covariance (ANCOVA) model with all but the^th predictor treated as nominal. Nominal variables are represented by a collection of binary variables indicating equality and inequality to the observed nominal categorical values for the variable.

ConfidenceLevel,VarianceEstimatorFunction, andWorkingPrecision are relevant to the computation of results after the initial fitting. These options can be set withinLinearModelFit to specify the default settings for results obtained from theFittedModel object. These options can also be set within an already constructedFittedModel object to override the option values originally given toLinearModelFit.

Here are the default and mean-squared error variance estimates:

IncludeConstantBasis,LinearOffsetFunction,NominalVariables, andWeights are relevant only to the fitting. Setting these options within an already constructedFittedModel object will have no further impact on the result.

A major feature of the model-fitting framework is the ability to obtain results after the fitting. The full list of available results can be obtained using"Properties".

This is the number of properties available for linear models:

The properties include basic information about the data, fitted model, and numerous results and diagnostics.

Properties related to data and the fitted function.

The"BestFitParameters" property gives the fitted parameter values{β₀,β₁,…}."BestFit" is the fitted function and"Function" gives the fitted function as a pure function."BasisFunctions" gives the list of functions, with being the constant 1 when a constant term is present in the model. The"DesignMatrix" is the design or model matrix for the data."Response" gives the list of the response or values from the original data.

Types of residuals.

Residuals give a measure of the pointwise difference between the fitted values and the original responses."FitResiduals" gives the differences between the observed and fitted values{y₁-,y₂-,…}."StandardizedResiduals" and"StudentizedResiduals" are scaled forms of the residuals. The^th standardized residual is, where is the estimated error variance, is the^th diagonal element of the hat matrix, and is the weight for the^th data point. The^th studentized residual uses the same formula with replaced by, the variance estimate omitting the^th data point.

Properties related to the sum of squared errors.

"ANOVATable" gives a formatted analysis of variance table for the model."ANOVATableEntries" gives the numeric entries in the table and the remainingANOVATable properties give the elements of columns in the table so individual parts of the table can easily be used in further computations.

This gives a formatted ANOVA table for the fitted model:

Here are the elements of the MS column of the table:

Properties and diagnostics for parameter estimates.

"CovarianceMatrix" gives the covariance between fitted parameters. The matrix is, where is the variance estimate, is the design matrix, and is the diagonal matrix of weights."CorrelationMatrix" is the associated correlation matrix for the parameter estimates."ParameterErrors" is equivalent to the square root of the diagonal elements of the covariance matrix.

"ParameterTable" and"ParameterConfidenceIntervalTable" contain information about the individual parameter estimates, tests of parameter significance, and confidence intervals.

Here is some data:

This fits a model using both predictor variables:

These are the formatted parameter and parameter confidence interval tables:

Here 99% confidence intervals are used in the table:

The Estimate column of these tables is equivalent to"BestFitParameters". The-statistics are the estimates divided by the standard errors. Each‐value is the two‐sided‐value for the-statistic and can be used to assess whether the parameter estimate is statistically significantly different from 0. Each confidence interval gives the upper and lower bounds for the parameter confidence interval at the level prescribed by theConfidenceLevel option. The variousParameterTable andParameterConfidenceIntervalTable properties can be used to get the columns or the unformatted array of values from the table.

"VarianceInflationFactors" is used to measure the multicollinearity between basis functions. The^th inflation factor is equal to, where is the coefficient of variation from fitting the^th basis function to a linear function of the other basis functions. WithIncludeConstantBasis->True, the first inflation factor is for the constant term.

"EigenstructureTable" gives the eigenvalues, condition indices, and variance partitions for the nonconstant basis functions. The Index column gives the square root of the ratios of the eigenvalues to the largest eigenvalue. The column for each basis function gives the proportion of variation in that basis function explained by the associated eigenvector."EigenstructureTablePartitions" gives the values in the variance partitioning for all basis functions in the table.

Properties related to influence measures.

Pointwise measures of influence are often employed to assess whether individual data points have a large impact on the fitting. The hat matrix and catcher matrix play important roles in such diagnostics. The hat matrix is the matrix such that, where is the observed response vector and is the predicted response vector."HatDiagonal" gives the diagonal elements of the hat matrix."CatcherMatrix" is the matrix such that, where is the fitted parameter vector.

"FitDifferences" gives the DFFITS values that provide a measure of influence of each data point on the fitted or predicted values. The^th DFFITS value is given by, where is the^th hat diagonal and is the^th studentized residual.

"BetaDifferences" gives the DFBETAS values that provide measures of influence of each data point on the parameters in the model. For a model with parameters, the^th element of"BetaDifferences" is a list of length with the^th value giving the measure of the influence of data point on the^th parameter in the model. The^th"BetaDifferences" vector can be written as, where is the,^th element of the catcher matrix.

"CookDistances" gives the Cook distance measures of leverage. The^th Cook distance is given by, where is the^th standardized residual.

The^th element of"CovarianceRatios" is given by and the^th"FVarianceRatios" value is equal to, where is the^th single deletion variance.

The Durbin–Watson‐statistic"DurbinWatsonD" is used for testing the existence of a first-order autoregressive process. The‐statistic is equivalent to, where is the^th residual.

This plots the Cook distances for the bivariate model:

Properties of predicted values.

Tabular results for confidence intervals are given by"MeanPredictionConfidenceIntervalTable" and"SinglePredictionConfidenceIntervalTable". These include the observed and predicted responses, standard error estimates, and confidence intervals for each point. Mean prediction confidence intervals are often referred to simply as confidence intervals and single prediction confidence intervals are often referred to as prediction intervals.

Mean prediction intervals give the confidence interval for the mean of the response at fixed values of the predictors and are given by, where is the quantile of the Student distribution with degrees of freedom, is the vector of basis functions evaluated at fixed predictors, and is the estimated covariance matrix for the parameters. Single prediction intervals provide the confidence interval for predicting at fixed values of the predictors, and are given by, where is the estimated error variance.

"MeanPredictionBands" and"SinglePredictionBands" give formulas for mean and single prediction confidence intervals as functions of the predictor variables.

Here is the mean prediction table:

This gives the 90% mean prediction intervals:

Goodness-of-fit measures.

Goodness-of-fit measures are used to assess how well a model fits or to compare models. The coefficient of determination"RSquared" is the ratio of the model sum of squares to the total sum of squares."AdjustedRSquared" penalizes for the number of parameters in the model and is given by.

"AIC" and"BIC" are likelihood‐based goodness-of-fit measures. Both are equal to times the log-likelihood for the model plus, where is the number of parameters to be estimated including the estimated variance. For"AIC" is, and for"BIC" is.

The linear model can be seen as a model with each response value being an observation from a normal distribution with mean value. The generalized linear model extends to models of the form, with each assumed to be an observation from a distribution of known exponential family form with mean, and being an invertible function over the support of the exponential family. Models of this sort can be obtained viaGeneralizedLinearModelFit.

Generalized linear model fitting.

The invertible function is called the link function and the linear combination is referred to as the linear predictor. Common special cases include the linear regression model with the identity link function and Gaussian or normal exponential family distribution, logit and probit models for probabilities, Poisson models for count data, and gamma and inverse Gaussian models.

The error variance is a function of the prediction and is defined by the distribution up to a constant, which is referred to as the dispersion parameter. The error variance for a fitted value can be written as, where is an estimate of the dispersion parameter obtained from the observed and predicted response values, and is the variance function associated with the exponential family evaluated at the value.

This fits a linear regression model:

This fits a canonical gamma regression model to the same data:

Here are the functional forms of the models:

Logit and probit models are common binomial models for probabilities. The link function for the logit model is and the link for the probit model is the inverse CDF for a standard normal distribution. Models of this type can be fitted viaGeneralizedLinearModelFit withExponentialFamily->"Binomial" and the appropriateLinkFunction or viaLogitModelFit andProbitModelFit.

Logit and probit model fitting.

Parameter estimates are obtained via iteratively reweighted least squares with weights obtained from the variance function of the assumed distribution. Options forGeneralizedLinearModelFit include options for iteration fitting such asPrecisionGoal, options for model specification such asLinkFunction, and options for further analysis such asConfidenceLevel.

Options forGeneralizedLinearModelFit.

The options forLogitModelFit andProbitModelFit are the same as forGeneralizedLinearModelFit except thatExponentialFamily andLinkFunction are defined by the logit or probit model and so are not options toLogitModelFit andProbitModelFit.

ExponentialFamily can be"Binomial","Gamma","Gaussian","InverseGaussian","Poisson", or"QuasiLikelihood". Binomial models are valid for responses from 0 to 1. Poisson models are valid for non-negative integer responses. Gaussian or normal models are valid for real responses. Gamma and inverse Gaussian models are valid for positive responses. Quasi-likelihood models define the distributional structure in terms of a variance function such that the log of the quasi‐likelihood function for the^th data point is given by. The variance function for a"QuasiLikelihood" model can be optionally set viaExponentialFamily->{"QuasiLikelihood", "VarianceFunction"->fun}, wherefun is a pure function to be applied to fitted values.

DispersionEstimatorFunction defines a function for estimating the dispersion parameter. The estimate is analogous to in linear and nonlinear regression models.

ExponentialFamily,IncludeConstantBasis,LinearOffsetFunction,LinkFunction,NominalVariables, andWeights all define some aspect of the model structure and optimization criterion and can only be set withinGeneralizedLinearModelFit. All other options can be set either withinGeneralizedLinearModelFit or passed to theFittedModel object when obtaining results and diagnostics. Options set in evaluations ofFittedModel objects take precedence over settings given toGeneralizedLinearModelFit at the time of the fitting.

This gives 95% and 99% confidence intervals for the parameters in the gamma model:

Properties related to data and the fitted function.

"BestFitParameters" gives the parameter estimates for the basis functions."BestFit" gives the fitted function, and"LinearPredictor" gives the linear combination."BasisFunctions" gives the list of functions, with being the constant 1 when a constant term is present in the model."DesignMatrix" is the design or model matrix for the basis functions.

Properties related to dispersion and model deviances.

Deviances and deviance tables generalize the model decomposition given by analysis of variance in linear models. The deviance for a single data point is, where is the log-likelihood function for the fitted model."Deviances" gives a list of the deviance values for all data points. The sum of all deviances gives the model deviance. The model deviance can be decomposed as sums of squares, which are in an ANOVA table for linear models. The full model is the model whose predicted values are the same as the data.

Here is some data with two predictor variables:

This fits the data to an inverse Gaussian model:

Here is the deviance table for the model:

As with sums of squares, deviances are additive. The Deviance column of the table gives the increase in the model deviance when the given basis function is added. The Residual Deviance column gives the difference between the model deviance and the deviance for the submodel containing all previous terms in the table. For large samples, the increase in deviance is approximately distributed with degrees of freedom equal to that for the basis function in the table.

"NullDeviance" is the deviance for the null model, the constant model equal to the mean of all observed responses for models including a constant, or if a constant term is not included.

As with"ANOVATable", a number of properties are included to extract the columns or unformatted array of entries from"DevianceTable".

Types of residuals.

"FitResiduals" is the list of residuals, differences between the observed and predicted responses. Given the distributional assumptions, the magnitude of the residuals is expected to change as a function of the predicted response value. Various types of scaled residuals are employed in the analysis of generalized linear models.

If and are the deviance and residual for the^th data point, the^th deviance residual is given by. The^th Pearson residual is defined as, where is the variance function for the exponential family distribution. Standardized deviance residuals and standardized Pearson residuals include division by, where is the^th diagonal of the hat matrix."LikelihoodResiduals" values combine deviance and Pearson residuals. The^th likelihood residual is given by.

"AnscombeResiduals" provide a transformation of the residuals toward normality, so a plot of these residuals should be expected to look roughly like white noise. The^th Anscombe residual can be written as.

"WorkingResiduals" gives the residuals from the last step of the iterative fitting. The^th working residual can be obtained as evaluated at.

This plots the residuals and Anscombe residuals for the inverse Gaussian model:

Properties and diagnostics for parameter estimates.

"CovarianceMatrix" gives the covariance between fitted parameters and is very similar to the definition for linear models. WithCovarianceEstimatorFunction->"ExpectedInformation", the expected information matrix obtained from the iterative fitting is used. The matrix is, where is the design matrix and is the diagonal matrix of weights from the final stage of the fitting. The weights include both weights specified via theWeights option and the weights associated with the distribution's variance function. WithCovarianceEstimatorFunction->"ObservedInformation", the matrix is given by, where is the observed Fisher information matrix, which is the Hessian of the log‐likelihood function with respect to parameters of the model.

"CorrelationMatrix" is the associated correlation matrix for the parameter estimates."ParameterErrors" is equivalent to the square root of the diagonal elements of the covariance matrix. "ParameterTable" and"ParameterConfidenceIntervalTable" contain information about the individual parameter estimates, tests of parameter significance, and confidence intervals. The test statistics for generalized linear models asymptotically follow normal distributions.

Properties related to influence measures.

"CookDistances" and"HatDiagonal" extend the leverage measures from linear regression to generalized linear models. The hat matrix from which the diagonal elements are extracted is defined using the final weights of the iterative fitting.

The Cook distance measures of leverage are defined as in linear regression with standardized residuals replaced by standardized Pearson residuals. The^th Cook distance is given by, where is the^th standardized Pearson residual.

Properties of predicted values.

Goodness-of-fit measures.

"LogLikelihood" is the log‐likelihood for the fitted model."AIC" and"BIC" are penalized log‐likelihood measures, where is the log‐likelihood for the fitted model, is the number of parameters estimated including the dispersion parameter, and is for"AIC" and for"BIC" for a model of data points."LikelihoodRatioStatistic" is given by, where is the log‐likelihood for the null model.

A number of the goodness-of-fit measures generalize from linear regression as either a measure of explained variation or as a likelihood‐based measure."CoxSnellPseudoRSquared" is given by."CraggUhlerPseudoRSquared" is a scaled version of Cox and Snell's measure."LikelihoodRatioIndex" involves the ratio of log‐likelihoods, and"AdjustedLikelihoodRatioIndex" adjusts by penalizing for the number of parameters."EfronPseudoRSquared" uses the sum of squares interpretation of and is given as, where is the^th residual and is the mean of the responses.

"PearsonChiSquare" is equal to, where the are Pearson residuals.

A nonlinear least-squares model is an extension of the linear model where the model need not be a linear combination of basis function. The errors are still assumed to be independent and normally distributed. Models of this type can be fitted using theNonlinearModelFit function.