Instatistics, ageneralized linear model (GLM) is a flexible generalization of ordinarylinear regression. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via alink function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.

Generalized linear models were formulated byJohn Nelder andRobert Wedderburn as a way of unifying various other statistical models, includinglinear regression,logistic regression andPoisson regression.^[1] They proposed aniteratively reweighted least squaresmethod formaximum likelihood estimation (MLE) of the model parameters. MLE remains popular and is the default method on many statistical computing packages. Other approaches, includingBayesian regression andleast squares fitting tovariance stabilized responses, have been developed.

Ordinary linear regression predicts theexpected value of a given unknown quantity (theresponse variable, arandom variable) as alinear combination of a set of observed values (predictors). This implies that a constant change in a predictor leads to a constant change in the response variable (i.e. alinear-response model). This is appropriate when the response variable can vary, to a good approximation, indefinitely in either direction, or more generally for any quantity that only varies by a relatively small amount compared to the variation in the predictive variables, e.g. human heights.

However, these assumptions are inappropriate for some types of response variables. For example, in cases where the response variable is expected to be always positive and varying over a wide range, constant input changes lead to geometrically (i.e. exponentially) varying, rather than constantly varying, output changes. As an example, suppose a linear prediction model learns from some data (perhaps primarily drawn from large beaches) that a 10 degree temperature decrease would lead to 1,000 fewer people visiting the beach. This model is unlikely to generalize well over differently-sized beaches. More specifically, the problem is that if the model is used to predict the new attendance with a temperature drop of 10 for a beach that regularly receives 50 beachgoers, it would predict an impossible attendance value of −950. Logically, a more realistic model would instead predict a constantrate of increased beach attendance (e.g. an increase of 10 degrees leads to a doubling in beach attendance, and a drop of 10 degrees leads to a halving in attendance). Such a model is termed anexponential-response model (orlog-linear model, since thelogarithm of the response is predicted to vary linearly).

Similarly, a model that predicts a probability of making a yes/no choice (aBernoulli variable) is even less suitable as a linear-response model, since probabilities are bounded on both ends (they must be between 0 and 1). Imagine, for example, a model that predicts the likelihood of a given person going to the beach as a function of temperature. A reasonable model might predict, for example, that a change in 10 degrees makes a person two times more or less likely to go to the beach. But what does "twice as likely" mean in terms of a probability? It cannot literally mean to double the probability value (e.g. 50% becomes 100%, 75% becomes 150%, etc.). Rather, it is theodds that are doubling: from 2:1 odds, to 4:1 odds, to 8:1 odds, etc. Such a model is alog-odds orlogistic model.

Generalized linear models cover all these situations by allowing for response variables that have arbitrary distributions (rather than simplynormal distributions), and for an arbitrary function of the response variable (thelink function) to vary linearly with the predictors (rather than assuming that the response itself must vary linearly). For example, the case above of predicted number of beach attendees would typically be modeled with aPoisson distribution and a log link, while the case of predicted probability of beach attendance would typically be modelled with aBernoulli distribution (orbinomial distribution, depending on exactly how the problem is phrased) and a log-odds (orlogit) link function.

In a generalized linear model (GLM), each outcomeY of thedependent variables is assumed to be generated from a particulardistribution in anexponential family, a large class ofprobability distributions that includes thenormal,binomial,Poisson andgamma distributions, among others. The conditional meanμ of the distribution depends on the independent variablesX through:

${\displaystyle \mathrm{E}(\mathbf{Y}\mid \mathbf{X})=\mathit{\mu }=g^{-1}(\mathbf{X}\mathit{\beta }),}$

where E(Y | X) is theexpected value ofYconditional onX;Xβ is thelinear predictor, a linear combination of unknown parametersβ;g is the link function.

In this framework, the variance is typically a function,V, of the mean:

${\displaystyle \mathrm{Var}(\mathbf{Y}\mid \mathbf{X})=\mathrm{V}(g^{-1}(\mathbf{X}\mathit{\beta })).}$

It is convenient ifV follows from an exponential family of distributions, but it may simply be that the variance is a function of the predicted value.

The unknown parameters,β, are typically estimated withmaximum likelihood, maximumquasi-likelihood, orBayesian techniques.

The GLM consists of three elements:

1. A particular distribution for modeling${\displaystyle Y}$ from among those which are considered exponential families of probability distributions,

2. A linear predictor${\displaystyle \eta =X\beta }$, and

3. A link function${\displaystyle g}$ such that${\displaystyle \mathrm{E}(Y\mid X)=\mu =g^{-1}(\eta )}$.

Anoverdispersed exponential family of distributions is a generalization of anexponential family and theexponential dispersion model of distributions and includes those families of probability distributions, parameterized by${\displaystyle \mathit{\theta }}$ and${\displaystyle \tau }$, whose density functionsf (orprobability mass function, for the case of adiscrete distribution) can be expressed in the form

${\displaystyle f_Y(\mathbf{y}\mid \mathit{\theta },\tau )=h(\mathbf{y},\tau )\exp \left(\frac{\mathbf{b}(\mathit{\theta }{)}^{\mathrm{T}}\mathbf{T}(\mathbf{y})-A(\mathit{\theta })}{d(\tau )}\right).}$

Thedispersion parameter,${\displaystyle \tau }$, typically is known and is usually related to the variance of the distribution. The functions${\displaystyle h(\mathbf{y},\tau )}$,${\displaystyle \mathbf{b}(\mathit{\theta })}$,${\displaystyle \mathbf{T}(\mathbf{y})}$,${\displaystyle A(\mathit{\theta })}$, and${\displaystyle d(\tau )}$ are known. Many common distributions are in this family, including the normal, exponential, gamma, Poisson, Bernoulli, and (for fixed number of trials) binomial, multinomial, and negative binomial.

For scalar${\displaystyle \mathbf{y}}$ and${\displaystyle \mathit{\theta }}$ (denoted${\displaystyle y}$ and${\displaystyle \theta }$ in this case), this reduces to

${\displaystyle f_Y(y\mid \theta ,\tau )=h(y,\tau )\exp \left(\frac{b(\theta )T(y)-A(\theta )}{d(\tau )}\right).}$

${\displaystyle \mathit{\theta }}$ is related to the mean of the distribution. If${\displaystyle \mathbf{b}(\mathit{\theta })}$ is the identity function, then the distribution is said to be incanonical form (ornatural form). Note that any distribution can be converted to canonical form by rewriting${\displaystyle \mathit{\theta }}$ as${\displaystyle {\mathit{\theta }}^{\prime }}$ and then applying the transformation${\displaystyle \mathit{\theta }=\mathbf{b}({\mathit{\theta }}^{\prime })}$. It is always possible to convert${\displaystyle A(\mathit{\theta })}$ in terms of the new parametrization, even if${\displaystyle \mathbf{b}({\mathit{\theta }}^{\prime })}$ is not aone-to-one function; see comments in the page onexponential families.

If, in addition,${\displaystyle \mathbf{T}(\mathbf{y})}$ and${\displaystyle \mathbf{b}(\mathit{\theta })}$ are the identity, then${\displaystyle \mathit{\theta }}$ is called thecanonical parameter (ornatural parameter) and is related to the mean through

${\displaystyle \mathit{\mu }=\mathrm{E}(\mathbf{y})={\mathrm{\nabla }}_{\mathit{\theta }}A(\mathit{\theta }).}$

For scalar${\displaystyle \mathbf{y}}$ and${\displaystyle \mathit{\theta }}$, this reduces to

${\displaystyle \mu =\mathrm{E}(y)=A^{\prime }(\theta ).}$

Under this scenario, the variance of the distribution can be shown to be^[2]

${\displaystyle \mathrm{Var}(\mathbf{y})={\mathrm{\nabla }}_{\mathit{\theta }}^{2}A(\mathit{\theta })d(\tau ).}$

For scalar${\displaystyle \mathbf{y}}$ and${\displaystyle \mathit{\theta }}$, this reduces to

${\displaystyle \mathrm{Var}(y)=A^{″}(\theta )d(\tau ).}$

The linear predictor is the quantity which incorporates the information about the independent variables into the model. The symbolη (Greek "eta") denotes a linear predictor. It is related to theexpected value of the data through the link function.

η is expressed as linear combinations (thus, "linear") of unknown parametersβ. The coefficients of the linear combination are represented as the matrix of independent variablesX.η can thus be expressed as

${\displaystyle \eta =\mathbf{X}\mathit{\beta }.}$

The link function provides the relationship between the linear predictor and themean of the distribution function. There are many commonly used link functions, and their choice is informed by several considerations. There is always a well-definedcanonical link function which is derived from the exponential of the response'sdensity function. However, in some cases it makes sense to try to match thedomain of the link function to therange of the distribution function's mean, or use a non-canonical link function for algorithmic purposes, for exampleBayesian probit regression.

When using a distribution function with a canonical parameter${\displaystyle \theta ,}$ the canonical link function is the function that expresses${\displaystyle \theta }$ in terms of${\displaystyle \mu ,}$ i.e.${\displaystyle \theta =g(\mu ).}$ For the most common distributions, the mean${\displaystyle \mu }$ is one of the parameters in the standard form of the distribution'sdensity function, and then${\displaystyle g(\mu )}$ is the function as defined above that maps the density function into its canonical form. When using the canonical link function,${\displaystyle g(\mu )=\theta =\mathbf{X}\mathit{\beta },}$ which allows${\displaystyle {\mathbf{X}}^{\mathrm{T}}\mathbf{Y}}$ to be asufficient statistic for${\displaystyle \mathit{\beta }}$.

Following is a table of several exponential-family distributions in common use and the data they are typically used for, along with the canonical link functions and their inverses (sometimes referred to as the mean function, as done here).

Common distributions with typical uses and canonical link functions

Distribution	Support of distribution	Typical uses	Link name	Link function,${\displaystyle \mathbf{X}\mathit{\beta }=g(\mu )}$	Mean function
Normal	real:${\displaystyle (-\mathrm{\infty },+\mathrm{\infty })}$	Linear-response data	Identity	${\displaystyle \mathbf{X}\mathit{\beta }=\mu }$	${\displaystyle \mu =\mathbf{X}\mathit{\beta }}$
Laplace
Exponential	real:${\displaystyle (0,+\mathrm{\infty })}$	Exponential-response data, scale parameters	Negative inverse	${\displaystyle \mathbf{X}\mathit{\beta }=-{\mu }^{-1}}$	${\displaystyle \mu =-(\mathbf{X}\mathit{\beta }{)}^{-1}}$
Gamma
Inverse Gaussian	real:${\displaystyle (0,+\mathrm{\infty })}$		Inverse squared	${\displaystyle \mathbf{X}\mathit{\beta }={\mu }^{-2}}$	${\displaystyle \mu =(\mathbf{X}\mathit{\beta }{)}^{-1/2}}$
Poisson	integer:${\displaystyle 0,1,2,\dots }$	count of occurrences in fixed amount of time/space	Log	${\displaystyle \mathbf{X}\mathit{\beta }=\ln (\mu )}$	${\displaystyle \mu =\exp (\mathbf{X}\mathit{\beta })}$
Bernoulli	integer:${\displaystyle \{0,1\}}$	outcome of single yes/no occurrence	Logit	${\displaystyle \mathbf{X}\mathit{\beta }=\ln \left(\frac{\mu }{1-\mu }\right)}$	${\displaystyle \mu =\frac{\exp (\mathbf{X}\mathit{\beta })}{1+\exp (\mathbf{X}\mathit{\beta })}=\frac{1}{1+\exp (-\mathbf{X}\mathit{\beta })}}$
Binomial	integer:${\displaystyle 0,1,\dots ,N}$	count of # of "yes" occurrences out of N yes/no occurrences		${\displaystyle \mathbf{X}\mathit{\beta }=\ln \left(\frac{\mu }{n-\mu }\right)}$
Categorical	integer:${\displaystyle [0,K)}$	outcome of singleK-way occurrence		${\displaystyle \mathbf{X}\mathit{\beta }=\ln \left(\frac{\mu }{1-\mu }\right)}$
	K-vector of integer:${\displaystyle [0,1]}$, where exactly one element in the vector has the value 1
Multinomial	K-vector of integer:${\displaystyle [0,N]}$	count of occurrences of different types (1, ...,K) out ofN totalK-way occurrences

In the cases of the exponential and gamma distributions, the domain of the canonical link function is not the same as the permitted range of the mean. In particular, the linear predictor may be positive, which would give an impossible negative mean. When maximizing the likelihood, precautions must be taken to avoid this. An alternative is to use a noncanonical link function.

In the case of the Bernoulli, binomial, categorical and multinomial distributions, the support of the distributions is not the same type of data as the parameter being predicted. In all of these cases, the predicted parameter is one or more probabilities, i.e. real numbers in the range${\displaystyle [0,1]}$. The resulting model is known aslogistic regression (ormultinomial logistic regression in the case thatK-way rather than binary values are being predicted).

For the Bernoulli and binomial distributions, the parameter is a single probability, indicating the likelihood of occurrence of a single event. The Bernoulli still satisfies the basic condition of the generalized linear model in that, even though a single outcome will always be either 0 or 1, theexpected value will nonetheless be a real-valued probability, i.e. the probability of occurrence of a "yes" (or 1) outcome. Similarly, in a binomial distribution, the expected value isNp, i.e. the expected proportion of "yes" outcomes will be the probability to be predicted.

For categorical and multinomial distributions, the parameter to be predicted is aK-vector of probabilities, with the further restriction that all probabilities must add up to 1. Each probability indicates the likelihood of occurrence of one of theK possible values. For the multinomial distribution, and for the vector form of the categorical distribution, the expected values of the elements of the vector can be related to the predicted probabilities similarly to the binomial and Bernoulli distributions.

Themaximum likelihood estimates can be found using aniteratively reweighted least squares algorithm or aNewton's method with updates of the form:

${\displaystyle {\mathit{\beta }}^{(t+1)}={\mathit{\beta }}^{(t)}+{\mathcal{J}}^{-1}({\mathit{\beta }}^{(t)})u({\mathit{\beta }}^{(t)}),}$

where${\displaystyle \mathcal{J}({\mathit{\beta }}^{(t)})}$ is theobserved information matrix (the negative of theHessian matrix) and${\displaystyle u({\mathit{\beta }}^{(t)})}$ is thescore function; or aFisher's scoring method:

${\displaystyle {\mathit{\beta }}^{(t+1)}={\mathit{\beta }}^{(t)}+{\mathcal{I}}^{-1}({\mathit{\beta }}^{(t)})u({\mathit{\beta }}^{(t)}),}$

where${\displaystyle \mathcal{I}({\mathit{\beta }}^{(t)})}$ is theFisher information matrix. Note that if the canonical link function is used, then they are the same.^[3]

In general, theposterior distribution cannot be found inclosed form and so must be approximated, usually usingLaplace approximations or some type ofMarkov chain Monte Carlo method such asGibbs sampling.

A possible point of confusion has to do with the distinction between generalized linear models andgeneral linear models, two broad statistical models. Co-originatorJohn Nelder has expressed regret over this terminology.^[4]

The general linear model may be viewed as a special case of the generalized linear model with identity link and responses normally distributed. As most exact results of interest are obtained only for the general linear model, the general linear model has undergone a somewhat longer historical development. Results for the generalized linear model with non-identity link areasymptotic (tending to work well with large samples).

A simple, very important example of a generalized linear model (also an example of a general linear model) islinear regression. In linear regression, the use of theleast-squares estimator is justified by theGauss–Markov theorem, which does not assume that the distribution is normal.

From the perspective of generalized linear models, however, it is useful to suppose that the distribution function is the normal distribution with constant variance and the link function is the identity, which is the canonical link if the variance is known. Under these assumptions, the least-squares estimator is obtained as the maximum-likelihood parameter estimate.

For the normal distribution, the generalized linear model has aclosed form expression for the maximum-likelihood estimates, which is convenient. Most other GLMs lackclosed form estimates.

When the response data,Y, are binary (taking on only values 0 and 1), the distribution function is generally chosen to be theBernoulli distribution and the interpretation ofμ_i is then the probability,p, ofY_i taking on the value one.

There are several popular link functions for binomial functions.

The most typical link function is the canonicallogit link:

${\displaystyle g(p)=\mathrm{logit}p=\ln \left(\frac{p}{1-p}\right).}$

GLMs with this setup arelogistic regression models (orlogit models).

Alternatively, the inverse of any continuouscumulative distribution function (CDF) can be used for the link since the CDF's range is${\displaystyle [0,1]}$, the range of the binomial mean. Thenormal CDF${\displaystyle \mathrm{\Phi }}$ is a popular choice and yields theprobit model. Its link is

${\displaystyle g(p)={\mathrm{\Phi }}^{-1}(p).}$

The reason for the use of the probit model is that a constant scaling of the input variable to a normal CDF (which can be absorbed through equivalent scaling of all of the parameters) yields a function that is practically identical to the logit function, but probit models are more tractable in some situations than logit models. (In a Bayesian setting in which normally distributedprior distributions are placed on the parameters, the relationship between the normal priors and the normal CDF link function means that aprobit model can be computed usingGibbs sampling, while a logit model generally cannot.)

The complementary log-log function may also be used:

${\displaystyle g(p)=\log (-\log (1-p)).}$

This link function is asymmetric and will often produce different results from the logit and probit link functions.^[5] The cloglog model corresponds to applications where we observe either zero events (e.g., defects) or one or more, where the number of events is assumed to follow thePoisson distribution.^[6] The Poisson assumption means that

${\displaystyle Pr(0)=\exp (-\mu ),}$

whereμ is a positive number denoting the expected number of events. Ifp represents the proportion of observations with at least one event, its complement

${\displaystyle 1-p=Pr(0)=\exp (-\mu ),}$

and then

${\displaystyle -\log (1-p)=\mu .}$

A linear model requires the response variable to take values over the entire real line. Sinceμ must be positive, we can enforce that by taking the logarithm, and letting log(μ) be a linear model. This produces the "cloglog" transformation

${\displaystyle \log (-\log (1-p))=\log (\mu ).}$

The identity linkg(p) = p is also sometimes used for binomial data to yield alinear probability model. However, the identity link can predict nonsense "probabilities" less than zero or greater than one. This can be avoided by using a transformation like cloglog, probit or logit (or any inverse cumulative distribution function). A primary merit of the identity link is that it can be estimated using linear math—and other standard link functions are approximately linear matching the identity link nearp = 0.5.

Thevariance function for "quasibinomial" data is:

${\displaystyle \mathrm{Var}(Y_i)=\tau {\mu }_i(1-{\mu }_i)}$

where the dispersion parameterτ is exactly 1 for the binomial distribution. Indeed, the standard binomial likelihood omitsτ. When it is present, the model is called "quasibinomial", and the modified likelihood is called aquasi-likelihood, since it is not generally the likelihood corresponding to any real family of probability distributions. Ifτ exceeds 1, the model is said to exhibitoverdispersion.

The binomial case may be easily extended to allow for amultinomial distribution as the response (also, a Generalized Linear Model for counts, with a constrained total). There are two ways in which this is usually done:

If the response variable isordinal, then one may fit a model function of the form:

${\displaystyle g({\mu }_m)={\eta }_m={\beta }_0+X_1{\beta }_1+\cdots +X_p{\beta }_p+{\gamma }_2+\cdots +{\gamma }_m={\eta }_1+{\gamma }_2+\cdots +{\gamma }_m\text{ where }{\mu }_m=\mathrm{P}(Y\le m).}$

form > 2. Different linksg lead toordinal regression models likeproportional odds models orordered probit models.

If the response variable is anominal measurement, or the data do not satisfy the assumptions of an ordered model, one may fit a model of the following form:

${\displaystyle g({\mu }_m)={\eta }_m={\beta }_{m,0}+X_1{\beta }_{m,1}+\cdots +X_p{\beta }_{m,p}\text{ where }{\mu }_m=\mathrm{P}(Y=m\mid Y\in \{1,m\}).}$

form > 2. Different linksg lead tomultinomial logit ormultinomial probit models. These are more general than the ordered response models, and more parameters are estimated.

Another example of generalized linear models includesPoisson regression which modelscount data using thePoisson distribution. The link is typically the logarithm, the canonical link.

The variance function is proportional to the mean

${\displaystyle \mathrm{var}(Y_i)=\tau {\mu }_i,}$

where the dispersion parameterτ is typically fixed at exactly one. When it is not, the resultingquasi-likelihood model is often described as Poisson withoverdispersion orquasi-Poisson.

The standard GLM assumes that the observations areuncorrelated. Extensions have been developed to allow forcorrelation between observations, as occurs for example inlongitudinal studies and clustered designs:

• Generalized estimating equations (GEEs) allow for the correlation between observations without the use of an explicit probability model for the origin of the correlations, so there is no explicitlikelihood. They are suitable when therandom effects and their variances are not of inherent interest, as they allow for the correlation without explaining its origin. The focus is on estimating the average response over the population ("population-averaged" effects) rather than the regression parameters that would enable prediction of the effect of changing one or more components ofX on a given individual. GEEs are usually used in conjunction withHuber–White standard errors.^[7][8]
• Generalized linear mixed models (GLMMs) are an extension to GLMs that includesrandom effects in the linear predictor, giving an explicit probability model that explains the origin of the correlations. The resulting "subject-specific" parameter estimates are suitable when the focus is on estimating the effect of changing one or more components ofX on a given individual. GLMMs are also referred to asmultilevel models and asmixed model. In general, fitting GLMMs is more computationally complex and intensive than fitting GEEs.

Generalized additive models (GAMs) are another extension to GLMs in which the linear predictorη is not restricted to be linear in the covariatesX but is the sum ofsmoothing functions applied to thex_is:

${\displaystyle \eta ={\beta }_0+f_1(x_1)+f_2(x_2)+\cdots }$

The smoothing functionsf_i are estimated from the data. In general this requires a large number of data points and is computationally intensive.^[9][10]

• Response modeling methodology
• Comparison of general and generalized linear models – Statistical linear modelPages displaying short descriptions of redirect targets
• Fractional model
• Generalized linear array model – model used for analyzing data sets with array structuresPages displaying wikidata descriptions as a fallback
• GLIM (software) – statistical software program for fitting generalized linear modelsPages displaying wikidata descriptions as a fallback
• Quasi-variance
• Natural exponential family – class of probability distributions that is a special case of an exponential familyPages displaying wikidata descriptions as a fallback
• Tweedie distribution – Family of probability distributions
• Variance functions – Smooth function in statisticsPages displaying short descriptions of redirect targets
• Vector generalized linear model (VGLM)
• Generalized estimating equation

• Dunn, P.K.; Smyth, G.K. (2018).Generalized Linear Models With Examples in R. New York: Springer.doi:10.1007/978-1-4419-0118-7.ISBN 978-1-4419-0118-7.{{cite book}}: CS1 maint: publisher location (link)
• Dobson, A.J.; Barnett, A.G. (2008).Introduction to Generalized Linear Models (3rd ed.). Boca Raton, FL: Chapman and Hall/CRC.ISBN 978-1-58488-165-0.
• Hardin, James;Hilbe, Joseph (2007).Generalized Linear Models and Extensions (2nd ed.). College Station: Stata Press.ISBN 978-1-59718-014-6.{{cite book}}: CS1 maint: publisher location (link)

• Media related toGeneralized linear models at Wikimedia Commons

• Actuarial science
• Generalized linear models
• Regression models

• Articles with short description
• Short description matches Wikidata
• Pages displaying short descriptions of redirect targets via Module:Annotated link
• Pages displaying wikidata descriptions as a fallback via Module:Annotated link
• CS1 maint: publisher location
• Commons category link is on Wikidata