Movatterモバイル変換

[0]ホーム
URL:

Jump to content
WikipediaThe Free Encyclopedia
Search

Logistic regression

From Wikipedia, the free encyclopedia
Statistical model for a binary dependent variable
"Logit model" redirects here; not to be confused withLogit function.
Example graph of a logistic regression curve fitted to data. The curve shows the estimated probability of passing an exam (binary dependent variable) versus hours studying (scalar independent variable). See§ Example for worked details.

Instatistics, alogistic model (orlogit model) is astatistical model that models thelog-odds of an event as alinear combination of one or moreindependent variables. Inregression analysis,logistic regression[1] (orlogit regression)estimates the parameters of a logistic model (the coefficients in the linear or non linear combinations). In binary logistic regression there is a singlebinarydependent variable, coded by anindicator variable, where the two values are labeled "0" and "1", while theindependent variables can each be a binary variable (two classes, coded by an indicator variable) or acontinuous variable (any real value). The corresponding probability of the value labeled "1" can vary between 0 (certainly the value "0") and 1 (certainly the value "1"), hence the labeling;[2] the function that converts log-odds to probability is thelogistic function, hence the name. Theunit of measurement for the log-odds scale is called alogit, fromlogistic unit, hence the alternative names. See§ Background and§ Definition for formal mathematics, and§ Example for a worked example.

Binary variables are widely used in statistics to model the probability of a certain class or event taking place, such as the probability of a team winning, of a patient being healthy, etc. (see§ Applications), and the logistic model has been the most commonly used model forbinary regression since about 1970.[3] Binary variables can be generalized tocategorical variables when there are more than two possible values (e.g. whether an image is of a cat, dog, lion, etc.), and the binary logistic regression generalized tomultinomial logistic regression. If the multiple categories areordered, one can use theordinal logistic regression (for example the proportional odds ordinal logistic model[4]). See§ Extensions for further extensions. The logistic regression model itself simply models probability of output in terms of input and does not performstatistical classification (it is not a classifier), though it can be used to make a classifier, for instance by choosing a cutoff value and classifying inputs with probability greater than the cutoff as one class, below the cutoff as the other; this is a common way to make abinary classifier.

Analogous linear models for binary variables with a differentsigmoid function instead of the logistic function (to convert the linear combination to a probability) can also be used, most notably theprobit model; see§ Alternatives. The defining characteristic of the logistic model is that increasing one of the independent variables multiplicatively scales the odds of the given outcome at aconstant rate, with each independent variable having its own parameter; for a binary dependent variable this generalizes theodds ratio. More abstractly, the logistic function is thenatural parameter for theBernoulli distribution, and in this sense is the "simplest" way to convert a real number to a probability.

The parameters of a logistic regression are most commonly estimated bymaximum-likelihood estimation (MLE). This does not have a closed-form expression, unlikelinear least squares; see§ Model fitting. Logistic regression by MLE plays a similarly basic role for binary or categorical responses as linear regression byordinary least squares (OLS) plays forscalar responses: it is a simple, well-analyzed baseline model; see§ Comparison with linear regression for discussion. The logistic regression as a general statistical model was originally developed and popularized primarily byJoseph Berkson,[5] beginning inBerkson (1944), where he coined "logit"; see§ History.

Part of a series on
Regression analysis
Models
Estimation
Background

Applications

[edit]

General

[edit]

Logistic regression is used in various fields, including machine learning, most medical fields, and social sciences. For example, the Trauma and Injury Severity Score (TRISS), which is widely used to predict mortality in injured patients, was originally developed by Boydet al. using logistic regression.[6] Many other medical scales used to assess severity of a patient have been developed using logistic regression.[7][8][9][10] Logistic regression may be used to predict the risk of developing a given disease (e.g.diabetes;coronary heart disease), based on observed characteristics of the patient (age, sex,body mass index, results of variousblood tests, etc.).[11][12] Another example might be to predict whether a Nepalese voter will vote Nepali Congress or Communist Party of Nepal or for any other party, based on age, income, sex, race, state of residence, votes in previous elections, etc.[13] The technique can also be used inengineering, especially for predicting the probability of failure of a given process, system or product.[14][15] It is also used inmarketing applications such as prediction of a customer's propensity to purchase a product or halt a subscription, etc.[16] Ineconomics, it can be used to predict the likelihood of a person ending up in the labor force, and a business application would be to predict the likelihood of a homeowner defaulting on amortgage.Conditional random fields, an extension of logistic regression to sequential data, are used innatural language processing. Disaster planners and engineers rely on these models to predict decisions taken by householders or building occupants in small-scale and large-scales evacuations, such as building fires, wildfires, hurricanes among others.[17][18][19] These models help in the development of reliabledisaster managing plans and safer design for thebuilt environment.

Supervised machine learning

[edit]

Logistic regression is asupervised machine learning algorithm widely used forbinary classification tasks, such as identifying whether an email is spam or not and diagnosing diseases by assessing the presence or absence of specific conditions based on patient test results. This approach utilizes the logistic (or sigmoid) function to transform a linear combination of input features into a probability value ranging between 0 and 1. This probability indicates the likelihood that a given input corresponds to one of two predefined categories. The essential mechanism of logistic regression is grounded in the logistic function's ability to model the probability of binary outcomes accurately. With its distinctive S-shaped curve, the logistic function effectively maps any real-valued number to a value within the 0 to 1 interval. This feature renders it particularly suitable for binary classification tasks, such as sorting emails into "spam" or "not spam". By calculating the probability that the dependent variable will be categorized into a specific group, logistic regression provides a probabilistic framework that supports informed decision-making.[20]

Example

[edit]

Problem

[edit]

As a simple example, we can use a logistic regression with one explanatory variable and two categories to answer the following question:

A group of 20 students spends between 0 and 6 hours studying for an exam. How does the number of hours spent studying affect the probability of the student passing the exam?

The reason for using logistic regression for this problem is that the values of the dependent variable, pass and fail, while represented by "1" and "0", are notcardinal numbers. If the problem was changed so that pass/fail was replaced with the grade 0–100 (cardinal numbers), then simpleregression analysis could be used.

The table shows the number of hours each student spent studying, and whether they passed (1) or failed (0).

Hours (xk)0.500.751.001.251.501.751.752.002.252.502.753.003.253.504.004.254.504.755.005.50
Pass (yk)00000010101010111111

We wish to fit a logistic function to the data consisting of the hours studied (xk) and the outcome of the test (yk =1 for pass, 0 for fail). The data points are indexed by the subscriptk which runs fromk=1{\displaystyle k=1} tok=K=20{\displaystyle k=K=20}. Thex variable is called the "explanatory variable", and they variable is called the "categorical variable" consisting of two categories: "pass" or "fail" corresponding to the categorical values 1 and 0 respectively.

Model

[edit]
Graph of a logistic regression curve fitted to the (xm,ym) data. The curve shows the probability of passing an exam versus hours studying.

Thelogistic function is of the form:

p(x)=11+e−(x−μ)/s{\displaystyle p(x)={\frac {1}{1+e^{-(x-\mu )/s}}}}

whereμ is alocation parameter (the midpoint of the curve, wherep(μ)=1/2{\displaystyle p(\mu )=1/2}) ands is ascale parameter. This expression may be rewritten as:

p(x)=11+e−(β0+β1x){\displaystyle p(x)={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x)}}}}

whereβ0=−μ/s{\displaystyle \beta _{0}=-\mu /s} and is known as theintercept (it is thevertical intercept ory-intercept of the liney=β0+β1x{\displaystyle y=\beta _{0}+\beta _{1}x}), andβ1=1/s{\displaystyle \beta _{1}=1/s} (inverse scale parameter orrate parameter): these are they-intercept and slope of the log-odds as a function ofx. Conversely,μ=−β0/β1{\displaystyle \mu =-\beta _{0}/\beta _{1}} ands=1/β1{\displaystyle s=1/\beta _{1}}.

Note that this model is actually an oversimplification, since it assumes everybody will pass if they learn long enough (limit = 1).

Fit

[edit]

The usual measure ofgoodness of fit for a logistic regression useslogistic loss (orlog loss), the negativelog-likelihood. For a givenxk andyk, writepk=p(xk){\displaystyle p_{k}=p(x_{k})}. The⁠pk{\displaystyle p_{k}}⁠ are the probabilities that the corresponding⁠yk{\displaystyle y_{k}}⁠ will equal one and⁠1−pk{\displaystyle 1-p_{k}}⁠ are the probabilities that they will be zero (seeBernoulli distribution). We wish to find the values of⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠ which give the "best fit" to the data. In the case of linear regression, the sum of the squared deviations of the fit from the data points (yk), thesquared error loss, is taken as a measure of the goodness of fit, and the best fit is obtained when that function isminimized.

The log loss for thek-th point⁠ℓk{\displaystyle \ell _{k}}⁠ is:

ℓk={−ln⁡pk if yk=1,−ln⁡(1−pk) if yk=0.{\displaystyle \ell _{k}={\begin{cases}-\ln p_{k}&{\text{ if }}y_{k}=1,\\-\ln(1-p_{k})&{\text{ if }}y_{k}=0.\end{cases}}}

The log loss can be interpreted as the "surprisal" of the actual outcome⁠yk{\displaystyle y_{k}}⁠ relative to the prediction⁠pk{\displaystyle p_{k}}⁠, and is a measure ofinformation content. Log loss is always greater than or equal to 0, equals 0 only in case of a perfect prediction (i.e., whenpk=1{\displaystyle p_{k}=1} andyk=1{\displaystyle y_{k}=1}, orpk=0{\displaystyle p_{k}=0} andyk=0{\displaystyle y_{k}=0}), and approaches infinity as the prediction gets worse (i.e., whenyk=1{\displaystyle y_{k}=1} andpk→0{\displaystyle p_{k}\to 0} oryk=0{\displaystyle y_{k}=0} andpk→1{\displaystyle p_{k}\to 1}), meaning the actual outcome is "more surprising". Since the value of the logistic function is always strictly between zero and one, the log loss is always greater than zero and less than infinity. Unlike in a linear regression, where the model can have zero loss at a point by passing through a data point (and zero loss overall if all points are on a line), in a logistic regression it is not possible to have zero loss at any points, since⁠yk{\displaystyle y_{k}}⁠ is either 0 or 1, but⁠0<pk<1{\displaystyle 0<p_{k}<1}⁠.

These can be combined into a single expression:

ℓk=−ykln⁡pk−(1−yk)ln⁡(1−pk).{\displaystyle \ell _{k}=-y_{k}\ln p_{k}-(1-y_{k})\ln(1-p_{k}).}

This expression is more formally known as thecross-entropy of the predicted distribution(pk,(1−pk)){\displaystyle {\big (}p_{k},(1-p_{k}){\big )}} from the actual distribution(yk,(1−yk)){\displaystyle {\big (}y_{k},(1-y_{k}){\big )}}, as probability distributions on the two-element space of (pass, fail).

The sum of these, the total loss, is the overall negative log-likelihood⁠−ℓ{\displaystyle -\ell }⁠, and the best fit is obtained for those choices of⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠ for which⁠−ℓ{\displaystyle -\ell }⁠ isminimized.

Alternatively, instead ofminimizing the loss, one canmaximize its inverse, the (positive) log-likelihood:

ℓ=∑k:yk=1ln⁡(pk)+∑k:yk=0ln⁡(1−pk)=∑k=1K(ykln⁡(pk)+(1−yk)ln⁡(1−pk)){\displaystyle \ell =\sum _{k:y_{k}=1}\ln(p_{k})+\sum _{k:y_{k}=0}\ln(1-p_{k})=\sum _{k=1}^{K}\left(\,y_{k}\ln(p_{k})+(1-y_{k})\ln(1-p_{k})\right)}

or equivalently maximize thelikelihood function itself, which is the probability that the given data set is produced by a particular logistic function:

L=∏k:yk=1pk∏k:yk=0(1−pk){\displaystyle L=\prod _{k:y_{k}=1}p_{k}\,\prod _{k:y_{k}=0}(1-p_{k})}

This method is known asmaximum likelihood estimation.

Parameter estimation

[edit]

Sinceℓ is nonlinear in⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠, determining their optimum values will require numerical methods. One method of maximizingℓ is to require the derivatives ofℓ with respect to⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠ to be zero:

0=∂ℓ∂β0=∑k=1K(yk−pk){\displaystyle 0={\frac {\partial \ell }{\partial \beta _{0}}}=\sum _{k=1}^{K}(y_{k}-p_{k})}
0=∂ℓ∂β1=∑k=1K(yk−pk)xk{\displaystyle 0={\frac {\partial \ell }{\partial \beta _{1}}}=\sum _{k=1}^{K}(y_{k}-p_{k})x_{k}}

and the maximization procedure can be accomplished by solving the above two equations for⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠, which, again, will generally require the use of numerical methods.

The values of⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠ which maximizeℓ andL using the above data are found to be:

β0≈−4.1{\displaystyle \beta _{0}\approx -4.1}
β1≈1.5{\displaystyle \beta _{1}\approx 1.5}

which yields a value forμ ands of:

μ=−β0/β1≈2.7{\displaystyle \mu =-\beta _{0}/\beta _{1}\approx 2.7}
s=1/β1≈0.67{\displaystyle s=1/\beta _{1}\approx 0.67}

Predictions

[edit]

The⁠β0{\displaystyle \beta _{0}}⁠ and⁠β1{\displaystyle \beta _{1}}⁠ coefficients may be entered into the logistic regression equation to estimate the probability of passing the exam.

For example, for a student who studies 2 hours, entering the valuex=2{\displaystyle x=2} into the equation gives the estimated probability of passing the exam of 0.25:

t=β0+2β1≈−4.1+2⋅1.5=−1.1{\displaystyle t=\beta _{0}+2\beta _{1}\approx -4.1+2\cdot 1.5=-1.1}
p=11+e−t≈0.25=Probability of passing exam{\displaystyle p={\frac {1}{1+e^{-t}}}\approx 0.25={\text{Probability of passing exam}}}

Similarly, for a student who studies 4 hours, the estimated probability of passing the exam is 0.87:

t=β0+4β1≈−4.1+4⋅1.5=1.9{\displaystyle t=\beta _{0}+4\beta _{1}\approx -4.1+4\cdot 1.5=1.9}
p=11+e−t≈0.87=Probability of passing exam{\displaystyle p={\frac {1}{1+e^{-t}}}\approx 0.87={\text{Probability of passing exam}}}

This table shows the estimated probability of passing the exam for several values of hours studying.

Hours
of study
(x)		Passing exam
Log-odds (t)Odds (et)Probability (p)
1−2.570.076 ≈ 1:13.10.07
2−1.070.34 ≈ 1:2.910.26
⁠μ≈2.7{\displaystyle \mu \approx 2.7}⁠01⁠1/2⁠ = 0.50
30.441.550.61
41.946.960.87
53.4531.40.97

Model evaluation

[edit]

The logistic regression analysis gives the following output.

CoefficientStd. Errorz-valuep-value (Wald)
Intercept (β0)−4.11.8−2.30.021
Hours (β1)1.50.91.70.017

By theWald test, the output indicates that hours studying is significantly associated with the probability of passing the exam (p=0.017{\displaystyle p=0.017}). Rather than the Wald method, the recommended method[21] to calculate thep-value for logistic regression is thelikelihood-ratio test (LRT), which for these data givep≈0.00064{\displaystyle p\approx 0.00064} (see§ Deviance and likelihood ratio tests below).

Generalizations

[edit]

This simple model is an example of binary logistic regression, and has one explanatory variable and a binary categorical variable which can assume one of two categorical values.Multinomial logistic regression is the generalization of binary logistic regression to include any number of explanatory variables and any number of categories.

Background

[edit]
Figure 1. The standard logistic functionσ(t){\displaystyle \sigma (t)};σ(t)∈(0,1){\displaystyle \sigma (t)\in (0,1)} for allt{\displaystyle t}.

Definition of the logistic function

[edit]

An explanation of logistic regression can begin with an explanation of the standardlogistic function. The logistic function is asigmoid function, which takes anyreal inputt{\displaystyle t}, and outputs a value between zero and one.[2] For the logit, this is interpreted as taking inputlog-odds and having outputprobability. Thestandard logistic functionσ:R→(0,1){\displaystyle \sigma :\mathbb {R} \rightarrow (0,1)} is defined as follows:

σ(t)=etet+1=11+e−t{\displaystyle \sigma (t)={\frac {e^{t}}{e^{t}+1}}={\frac {1}{1+e^{-t}}}}

A graph of the logistic function on thet-interval (−6,6) is shown in Figure 1.

Let us assume thatt{\displaystyle t} is a linear function of a singleexplanatory variablex{\displaystyle x} (the case wheret{\displaystyle t} is alinear combination of multiple explanatory variables is treated similarly). We can then expresst{\displaystyle t} as follows:

t=β0+β1x{\displaystyle t=\beta _{0}+\beta _{1}x}

And the general logistic functionp:R→(0,1){\displaystyle p:\mathbb {R} \rightarrow (0,1)} can now be written as:

p(x)=σ(t)=11+e−(β0+β1x){\displaystyle p(x)=\sigma (t)={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x)}}}}

In the logistic model,p(x){\displaystyle p(x)} is interpreted as the probability of the dependent variableY{\displaystyle Y} equaling a success/case rather than a failure/non-case. It is clear that theresponse variablesYi{\displaystyle Y_{i}} are not identically distributed:P(Yi=1∣X){\displaystyle P(Y_{i}=1\mid X)} differs from one data pointXi{\displaystyle X_{i}} to another, though they are independent givendesign matrixX{\displaystyle X} and shared parametersβ{\displaystyle \beta }.[11]

Definition of the inverse of the logistic function

[edit]

We can now define thelogit (log odds) function as the inverseg=σ−1{\displaystyle g=\sigma ^{-1}} of the standard logistic function. It is easy to see that it satisfies:

g(p(x))=σ−1(p(x))=logit⁡p(x)=ln⁡(p(x)1−p(x))=β0+β1x,{\displaystyle g(p(x))=\sigma ^{-1}(p(x))=\operatorname {logit} p(x)=\ln \left({\frac {p(x)}{1-p(x)}}\right)=\beta _{0}+\beta _{1}x,}

and equivalently, after exponentiating both sides we have the odds:

p(x)1−p(x)=eβ0+β1x.{\displaystyle {\frac {p(x)}{1-p(x)}}=e^{\beta _{0}+\beta _{1}x}.}

Interpretation of these terms

[edit]

In the above equations, the terms are as follows:

  • g{\displaystyle g} is the logit function. The equation forg(p(x)){\displaystyle g(p(x))} illustrates that thelogit (i.e., log-odds or natural logarithm of the odds) is equivalent to the linear regression expression.
  • ln{\displaystyle \ln } denotes thenatural logarithm.
  • p(x){\displaystyle p(x)} is the probability that the dependent variable equals a case, given some linear combination of the predictors. The formula forp(x){\displaystyle p(x)} illustrates that the probability of the dependent variable equaling a case is equal to the value of the logistic function of the linear regression expression. This is important in that it shows that the value of the linear regression expression can vary from negative to positive infinity and yet, after transformation, the resulting expression for the probabilityp(x){\displaystyle p(x)} ranges between 0 and 1.
  • β0{\displaystyle \beta _{0}} is theintercept from the linear regression equation (the value of the criterion when the predictor is equal to zero).
  • β1x{\displaystyle \beta _{1}x} is the regression coefficient multiplied by some value of the predictor.
  • basee{\displaystyle e} denotes the exponential function.

Definition of the odds

[edit]

The odds of the dependent variable equaling a case (given some linear combinationx{\displaystyle x} of the predictors) is equivalent to the exponential function of the linear regression expression. This illustrates how thelogit serves as a link function between the probability and the linear regression expression. Given that the logit ranges between negative and positive infinity, it provides an adequate criterion upon which to conduct linear regression and the logit is easily converted back into the odds.[2]

So we define odds of the dependent variable equaling a case (given some linear combinationx{\displaystyle x} of the predictors) as follows:

odds=eβ0+β1x.{\displaystyle {\text{odds}}=e^{\beta _{0}+\beta _{1}x}.}

The odds ratio

[edit]

For a continuous independent variable the odds ratio can be defined as:

The image represents an outline of what an odds ratio looks like in writing, through a template in addition to the test score example in the "Example" section of the contents. In simple terms, if we hypothetically get an odds ratio of 2 to 1, we can say... "For every one-unit increase in hours studied, the odds of passing (group 1) or failing (group 0) are (expectedly) 2 to 1 (Denis, 2019).
OR=odds⁡(x+1)odds⁡(x)=(p(x+1)1−p(x+1))(p(x)1−p(x))=eβ0+β1(x+1)eβ0+β1x=eβ1{\displaystyle \mathrm {OR} ={\frac {\operatorname {odds} (x+1)}{\operatorname {odds} (x)}}={\frac {\left({\frac {p(x+1)}{1-p(x+1)}}\right)}{\left({\frac {p(x)}{1-p(x)}}\right)}}={\frac {e^{\beta _{0}+\beta _{1}(x+1)}}{e^{\beta _{0}+\beta _{1}x}}}=e^{\beta _{1}}}

This exponential relationship provides an interpretation forβ1{\displaystyle \beta _{1}}: The odds multiply byeβ1{\displaystyle e^{\beta _{1}}} for every 1-unit increase in x.[22]

For a binary independent variable the odds ratio is defined asadbc{\displaystyle {\frac {ad}{bc}}} wherea,b,c andd are cells in a 2×2contingency table.[23]

Multiple explanatory variables

[edit]

If there are multiple explanatory variables, the above expressionβ0+β1x{\displaystyle \beta _{0}+\beta _{1}x} can be revised toβ0+β1x1+β2x2+⋯+βmxm=β0+∑i=1mβixi{\displaystyle \beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m}=\beta _{0}+\sum _{i=1}^{m}\beta _{i}x_{i}}. Then when this is used in the equation relating the log odds of a success to the values of the predictors, the linear regression will be amultiple regression withm explanators; the parametersβi{\displaystyle \beta _{i}} for alli=0,1,2,…,m{\displaystyle i=0,1,2,\dots ,m} are all estimated.

Again, the more traditional equations are:

log⁡p1−p=β0+β1x1+β2x2+⋯+βmxm{\displaystyle \log {\frac {p}{1-p}}=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m}}

and

p=11+b−(β0+β1x1+β2x2+⋯+βmxm){\displaystyle p={\frac {1}{1+b^{-(\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m})}}}}

where usuallyb=e{\displaystyle b=e}.

Definition

[edit]

A dataset containsN points. Each pointi consists of a set ofm input variablesx1,i ...xm,i (also calledindependent variables, explanatory variables, predictor variables, features, or attributes), and abinary outcome variableYi (also known as adependent variable, response variable, output variable, or class), i.e. it can assume only the two possible values 0 (often meaning "no" or "failure") or 1 (often meaning "yes" or "success"). The goal of logistic regression is to use the dataset to create a predictive model of the outcome variable.

As in linear regression, the outcome variablesYi are assumed to depend on the explanatory variablesx1,i ...xm,i.

Explanatory variables

The explanatory variables may be of anytype:real-valued,binary,categorical, etc. The main distinction is betweencontinuous variables anddiscrete variables.

(Discrete variables referring to more than two possible choices are typically coded usingdummy variables (orindicator variables), that is, separate explanatory variables taking the value 0 or 1 are created for each possible value of the discrete variable, with a 1 meaning "variable does have the given value" and a 0 meaning "variable does not have that value".)

Outcome variables

Formally, the outcomesYi are described as beingBernoulli-distributed data, where each outcome is determined by an unobserved probabilitypi that is specific to the outcome at hand, but related to the explanatory variables. This can be expressed in any of the following equivalent forms:

Yi∣x1,i,…,xm,i ∼Bernoulli⁡(pi)E⁡[Yi∣x1,i,…,xm,i]=piPr(Yi=y∣x1,i,…,xm,i)={piif y=11−piif y=0Pr(Yi=y∣x1,i,…,xm,i)=piy(1−pi)(1−y){\displaystyle {\begin{aligned}Y_{i}\mid x_{1,i},\ldots ,x_{m,i}\ &\sim \operatorname {Bernoulli} (p_{i})\\[5pt]\operatorname {\mathbb {E} } [Y_{i}\mid x_{1,i},\ldots ,x_{m,i}]&=p_{i}\\[5pt]\Pr(Y_{i}=y\mid x_{1,i},\ldots ,x_{m,i})&={\begin{cases}p_{i}&{\text{if }}y=1\\1-p_{i}&{\text{if }}y=0\end{cases}}\\[5pt]\Pr(Y_{i}=y\mid x_{1,i},\ldots ,x_{m,i})&=p_{i}^{y}(1-p_{i})^{(1-y)}\end{aligned}}}

The meanings of these four lines are:

  1. The first line expresses theprobability distribution of eachYi : conditioned on the explanatory variables, it follows aBernoulli distribution with parameterspi, the probability of the outcome of 1 for triali. As noted above, each separate trial has its own probability of success, just as each trial has its own explanatory variables. The probability of successpi is not observed, only the outcome of an individual Bernoulli trial using that probability.
  2. The second line expresses the fact that theexpected value of eachYi is equal to the probability of successpi, which is a general property of the Bernoulli distribution. In other words, if we run a large number of Bernoulli trials using the same probability of successpi, then take the average of all the 1 and 0 outcomes, then the result would be close topi. This is because doing an average this way simply computes the proportion of successes seen, which we expect to converge to the underlying probability of success.
  3. The third line writes out theprobability mass function of the Bernoulli distribution, specifying the probability of seeing each of the two possible outcomes.
  4. The fourth line is another way of writing the probability mass function, which avoids having to write separate cases and is more convenient for certain types of calculations. This relies on the fact thatYi can take only the value 0 or 1. In each case, one of the exponents will be 1, "choosing" the value under it, while the other is 0, "canceling out" the value under it. Hence, the outcome is eitherpi or 1 − pi, as in the previous line.
Linear predictor function

The basic idea of logistic regression is to use the mechanism already developed forlinear regression by modeling the probabilitypi using alinear predictor function, i.e. alinear combination of the explanatory variables and a set ofregression coefficients that are specific to the model at hand but the same for all trials. The linear predictor functionf(i){\displaystyle f(i)} for a particular data pointi is written as:

f(i)=β0+β1x1,i+⋯+βmxm,i,{\displaystyle f(i)=\beta _{0}+\beta _{1}x_{1,i}+\cdots +\beta _{m}x_{m,i},}

whereβ0,…,βm{\displaystyle \beta _{0},\ldots ,\beta _{m}} areregression coefficients indicating the relative effect of a particular explanatory variable on the outcome.

The model is usually put into a more compact form as follows:

  • The regression coefficientsβ0,β1, ...,βm are grouped into a single vectorβ of sizem + 1.
  • For each data pointi, an additional explanatory pseudo-variablex0,i is added, with a fixed value of 1, corresponding to theintercept coefficientβ0.
  • The resulting explanatory variablesx0,i,x1,i, ...,xm,i are then grouped into a single vectorXi of sizem + 1.

This makes it possible to write the linear predictor function as follows:

f(i)=β⋅Xi,{\displaystyle f(i)={\boldsymbol {\beta }}\cdot \mathbf {X} _{i},}

using the notation for adot product between two vectors.

This is an example of an SPSS output for a logistic regression model using three explanatory variables (coffee use per week, energy drink use per week, and soda use per week) and two categories (male and female).

Many explanatory variables, two categories

[edit]

The above example of binary logistic regression on one explanatory variable can be generalized to binary logistic regression on any number of explanatory variablesx1, x2,... and any number of categorical valuesy=0,1,2,…{\displaystyle y=0,1,2,\dots }.

To begin with, we may consider a logistic model withM explanatory variables,x1,x2 ...xM and, as in the example above, two categorical values (y = 0 and 1). For the simple binary logistic regression model, we assumed alinear relationship between the predictor variable and the log-odds (also calledlogit) of the event thaty=1{\displaystyle y=1}. This linear relationship may be extended to the case ofM explanatory variables:

t=logb⁡p1−p=β0+β1x1+β2x2+⋯+βMxM{\displaystyle t=\log _{b}{\frac {p}{1-p}}=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{M}x_{M}}

wheret is the log-odds andβi{\displaystyle \beta _{i}} are parameters of the model. An additional generalization has been introduced in which the base of the model (b) is not restricted toEuler's numbere. In most applications, the baseb{\displaystyle b} of the logarithm is usually taken to bee. However, in some cases it can be easier to communicate results by working in base 2 or base 10.

For a more compact notation, we will specify the explanatory variables and theβ coefficients as⁠(M+1){\displaystyle (M+1)}⁠-dimensional vectors:

x={x0,x1,x2,…,xM}{\displaystyle {\boldsymbol {x}}=\{x_{0},x_{1},x_{2},\dots ,x_{M}\}}
β={β0,β1,β2,…,βM}{\displaystyle {\boldsymbol {\beta }}=\{\beta _{0},\beta _{1},\beta _{2},\dots ,\beta _{M}\}}

with an added explanatory variablex0 =1. The logit may now be written as:

t=∑m=0Mβmxm=β⋅x{\displaystyle t=\sum _{m=0}^{M}\beta _{m}x_{m}={\boldsymbol {\beta }}\cdot x}

Solving for the probabilityp thaty=1{\displaystyle y=1} yields:

p(x)=bβ⋅x1+bβ⋅x=11+b−β⋅x=Sb(t){\displaystyle p({\boldsymbol {x}})={\frac {b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}{1+b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}}={\frac {1}{1+b^{-{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}}=S_{b}(t)},

whereSb{\displaystyle S_{b}} is thesigmoid function with baseb{\displaystyle b}. The above formula shows that once theβm{\displaystyle \beta _{m}} are fixed, we can easily compute either the log-odds thaty=1{\displaystyle y=1} for a given observation, or the probability thaty=1{\displaystyle y=1} for a given observation. The main use-case of a logistic model is to be given an observationx{\displaystyle {\boldsymbol {x}}}, and estimate the probabilityp(x){\displaystyle p({\boldsymbol {x}})} thaty=1{\displaystyle y=1}. The optimum beta coefficients may again be found by maximizing the log-likelihood. ForK measurements, definingxk{\displaystyle {\boldsymbol {x}}_{k}} as the explanatory vector of thek-th measurement, andyk{\displaystyle y_{k}} as the categorical outcome of that measurement, the log likelihood may be written in a form very similar to the simpleM=1{\displaystyle M=1} case above:

ℓ=∑k=1Kyklogb⁡(p(xk))+∑k=1K(1−yk)logb⁡(1−p(xk)){\displaystyle \ell =\sum _{k=1}^{K}y_{k}\log _{b}(p({\boldsymbol {x_{k}}}))+\sum _{k=1}^{K}(1-y_{k})\log _{b}(1-p({\boldsymbol {x_{k}}}))}

As in the simple example above, finding the optimumβ parameters will require numerical methods. One useful technique is to equate the derivatives of the log likelihood with respect to each of theβ parameters to zero yielding a set of equations which will hold at the maximum of the log likelihood:

∂ℓ∂βm=0=∑k=1Kykxmk−∑k=1Kp(xk)xmk{\displaystyle {\frac {\partial \ell }{\partial \beta _{m}}}=0=\sum _{k=1}^{K}y_{k}x_{mk}-\sum _{k=1}^{K}p({\boldsymbol {x}}_{k})x_{mk}}

wherexmk is the value of thexm explanatory variable from thek-th measurement.

Consider an example withM=2{\displaystyle M=2} explanatory variables,b=10{\displaystyle b=10}, and coefficientsβ0=−3{\displaystyle \beta _{0}=-3},β1=1{\displaystyle \beta _{1}=1}, andβ2=2{\displaystyle \beta _{2}=2} which have been determined by the above method. To be concrete, the model is:

t=log10⁡p1−p=−3+x1+2x2{\displaystyle t=\log _{10}{\frac {p}{1-p}}=-3+x_{1}+2x_{2}}
p=bβ⋅x1+bβ⋅x=bβ0+β1x1+β2x21+bβ0+β1x1+β2x2=11+b−(β0+β1x1+β2x2){\displaystyle p={\frac {b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}{1+b^{{\boldsymbol {\beta }}\cdot x}}}={\frac {b^{\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}}}{1+b^{\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}}}}={\frac {1}{1+b^{-(\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2})}}}},

wherep is the probability of the event thaty=1{\displaystyle y=1}. This can be interpreted as follows:

Multinomial logistic regression: Many explanatory variables and many categories

[edit]
Main article:Multinomial logistic regression

In the above cases of two categories (binomial logistic regression), the categories were indexed by "0" and "1", and we had two probabilities: The probability that the outcome was in category 1 was given byp(x){\displaystyle p({\boldsymbol {x}})}and the probability that the outcome was in category 0 was given by1−p(x){\displaystyle 1-p({\boldsymbol {x}})}. The sum of these probabilities equals 1, which must be true, since "0" and "1" are the only possible categories in this setup.

In general, if we have⁠M+1{\displaystyle M+1}⁠ explanatory variables (includingx0) and⁠N+1{\displaystyle N+1}⁠ categories, we will need⁠N+1{\displaystyle N+1}⁠ separate probabilities, one for each category, indexed byn, which describe the probability that the categorical outcomey will be in categoryy=n, conditional on the vector of covariatesx. The sum of these probabilities over all categories must equal 1. Using the mathematically convenient basee, these probabilities are:

pn(x)=eβn⋅x1+∑u=1Neβu⋅x{\displaystyle p_{n}({\boldsymbol {x}})={\frac {e^{{\boldsymbol {\beta }}_{n}\cdot {\boldsymbol {x}}}}{1+\sum _{u=1}^{N}e^{{\boldsymbol {\beta }}_{u}\cdot {\boldsymbol {x}}}}}} forn=1,2,…,N{\displaystyle n=1,2,\dots ,N}
p0(x)=1−∑n=1Npn(x)=11+∑u=1Neβu⋅x{\displaystyle p_{0}({\boldsymbol {x}})=1-\sum _{n=1}^{N}p_{n}({\boldsymbol {x}})={\frac {1}{1+\sum _{u=1}^{N}e^{{\boldsymbol {\beta }}_{u}\cdot {\boldsymbol {x}}}}}}

Each of the probabilities exceptp0(x){\displaystyle p_{0}({\boldsymbol {x}})} will have their own set of regression coefficientsβn{\displaystyle {\boldsymbol {\beta }}_{n}}. It can be seen that, as required, the sum of thepn(x){\displaystyle p_{n}({\boldsymbol {x}})} over all categoriesn is 1. The selection ofp0(x){\displaystyle p_{0}({\boldsymbol {x}})} to be defined in terms of the other probabilities is artificial. Any of the probabilities could have been selected to be so defined. This special value ofn is termed the "pivot index", and the log-odds (tn) are expressed in terms of the pivot probability and are again expressed as a linear combination of the explanatory variables:

tn=ln⁡(pn(x)p0(x))=βn⋅x{\displaystyle t_{n}=\ln \left({\frac {p_{n}({\boldsymbol {x}})}{p_{0}({\boldsymbol {x}})}}\right)={\boldsymbol {\beta }}_{n}\cdot {\boldsymbol {x}}}

Note also that for the simple case ofN=1{\displaystyle N=1}, the two-category case is recovered, withp(x)=p1(x){\displaystyle p({\boldsymbol {x}})=p_{1}({\boldsymbol {x}})} andp0(x)=1−p1(x){\displaystyle p_{0}({\boldsymbol {x}})=1-p_{1}({\boldsymbol {x}})}.

The log-likelihood that a particular set ofK measurements or data points will be generated by the above probabilities can now be calculated. Indexing each measurement byk, let thek-th set of measured explanatory variables be denoted byxk{\displaystyle {\boldsymbol {x}}_{k}} and their categorical outcomes be denoted byyk{\displaystyle y_{k}} which can be equal to any integer in [0,N]. The log-likelihood is then:

ℓ=∑k=1K∑n=0NΔ(n,yk)ln⁡(pn(xk)){\displaystyle \ell =\sum _{k=1}^{K}\sum _{n=0}^{N}\Delta (n,y_{k})\,\ln(p_{n}({\boldsymbol {x}}_{k}))}

whereΔ(n,yk){\displaystyle \Delta (n,y_{k})} is anindicator function which equals 1 ifyk = n and zero otherwise. In the case of two explanatory variables, this indicator function was defined asyk whenn = 1 and1-yk whenn = 0. This was convenient, but not necessary.[24] Again, the optimum beta coefficients may be found by maximizing the log-likelihood function generally using numerical methods. A possible method of solution is to set the derivatives of the log-likelihood with respect to each beta coefficient equal to zero and solve for the beta coefficients:

∂ℓ∂βnm=0=∑k=1KΔ(n,yk)xmk−∑k=1Kpn(xk)xmk{\displaystyle {\frac {\partial \ell }{\partial \beta _{nm}}}=0=\sum _{k=1}^{K}\Delta (n,y_{k})x_{mk}-\sum _{k=1}^{K}p_{n}({\boldsymbol {x}}_{k})x_{mk}}

whereβnm{\displaystyle \beta _{nm}} is them-th coefficient of theβn{\displaystyle {\boldsymbol {\beta }}_{n}} vector andxmk{\displaystyle x_{mk}} is them-th explanatory variable of thek-th measurement. Once the beta coefficients have been estimated from the data, we will be able to estimate the probability that any subsequent set of explanatory variables will result in any of the possible outcome categories.

Interpretations

[edit]

There are various equivalent specifications and interpretations of logistic regression, which fit into different types of more general models, and allow different generalizations.

As a generalized linear model

[edit]

The particular model used by logistic regression, which distinguishes it from standardlinear regression and from other types ofregression analysis used forbinary-valued outcomes, is the way the probability of a particular outcome is linked to the linear predictor function:

logit⁡(E⁡[Yi∣x1,i,…,xm,i])=logit⁡(pi)=ln⁡(pi1−pi)=β0+β1x1,i+⋯+βmxm,i{\displaystyle \operatorname {logit} (\operatorname {\mathbb {E} } [Y_{i}\mid x_{1,i},\ldots ,x_{m,i}])=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)=\beta _{0}+\beta _{1}x_{1,i}+\cdots +\beta _{m}x_{m,i}}

Written using the more compact notation described above, this is:

logit⁡(E⁡[Yi∣Xi])=logit⁡(pi)=ln⁡(pi1−pi)=β⋅Xi{\displaystyle \operatorname {logit} (\operatorname {\mathbb {E} } [Y_{i}\mid \mathbf {X} _{i}])=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}

This formulation expresses logistic regression as a type ofgeneralized linear model, which predicts variables with various types ofprobability distributions by fitting a linear predictor function of the above form to some sort of arbitrary transformation of the expected value of the variable.

The intuition for transforming using the logit function (the natural log of the odds) was explained above[clarification needed]. It also has the practical effect of converting the probability (which is bounded to be between 0 and 1) to a variable that ranges over(−∞,+∞){\displaystyle (-\infty ,+\infty )} — thereby matching the potential range of the linear prediction function on the right side of the equation.

Both the probabilitiespi and the regression coefficients are unobserved, and the means of determining them is not part of the model itself. They are typically determined by some sort of optimization procedure, e.g.maximum likelihood estimation, that finds values that best fit the observed data (i.e. that give the most accurate predictions for the data already observed), usually subject toregularization conditions that seek to exclude unlikely values, e.g. extremely large values for any of the regression coefficients. The use of a regularization condition is equivalent to doingmaximum a posteriori (MAP) estimation, an extension of maximum likelihood. (Regularization is most commonly done usinga squared regularizing function, which is equivalent to placing a zero-meanGaussianprior distribution on the coefficients, but other regularizers are also possible.) Whether or not regularization is used, it is usually not possible to find a closed-form solution; instead, an iterative numerical method must be used, such asiteratively reweighted least squares (IRLS) or, more commonly these days, aquasi-Newton method such as theL-BFGS method.[25]

The interpretation of theβj parameter estimates is as the additive effect on the log of theodds for a unit change in thej the explanatory variable. In the case of a dichotomous explanatory variable, for instance, gendereβ{\displaystyle e^{\beta }} is the estimate of the odds of having the outcome for, say, males compared with females.

An equivalent formula uses the inverse of the logit function, which is thelogistic function, i.e.:

E⁡[Yi∣Xi]=pi=logit−1⁡(β⋅Xi)=11+e−β⋅Xi{\displaystyle \operatorname {\mathbb {E} } [Y_{i}\mid \mathbf {X} _{i}]=p_{i}=\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})={\frac {1}{1+e^{-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}}

The formula can also be written as aprobability distribution (specifically, using aprobability mass function):

Pr(Yi=y∣Xi)=piy(1−pi)1−y=(eβ⋅Xi1+eβ⋅Xi)y(1−eβ⋅Xi1+eβ⋅Xi)1−y=eβ⋅Xi⋅y1+eβ⋅Xi{\displaystyle \Pr(Y_{i}=y\mid \mathbf {X} _{i})={p_{i}}^{y}(1-p_{i})^{1-y}=\left({\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{y}\left(1-{\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{1-y}={\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}\cdot y}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}}

As a latent-variable model

[edit]

The logistic model has an equivalent formulation as alatent-variable model. This formulation is common in the theory ofdiscrete choice models and makes it easier to extend to certain more complicated models with multiple, correlated choices, as well as to compare logistic regression to the closely relatedprobit model.

Imagine that, for each triali, there is a continuouslatent variableYi* (i.e. an unobservedrandom variable) that is distributed as follows:

Yi∗=β⋅Xi+εi{\displaystyle Y_{i}^{\ast }={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon _{i}\,}

where

εi∼Logistic⁡(0,1){\displaystyle \varepsilon _{i}\sim \operatorname {Logistic} (0,1)\,}

i.e. the latent variable can be written directly in terms of the linear predictor function and an additive randomerror variable that is distributed according to a standardlogistic distribution.

ThenYi can be viewed as an indicator for whether this latent variable is positive:

Yi={1if Yi∗>0  i.e. −εi<β⋅Xi,0otherwise.{\displaystyle Y_{i}={\begin{cases}1&{\text{if }}Y_{i}^{\ast }>0\ {\text{ i.e. }}{-\varepsilon _{i}}<{\boldsymbol {\beta }}\cdot \mathbf {X} _{i},\\0&{\text{otherwise.}}\end{cases}}}

The choice of modeling the error variable specifically with a standard logistic distribution, rather than a general logistic distribution with the location and scale set to arbitrary values, seems restrictive, but in fact, it is not. It must be kept in mind that we can choose the regression coefficients ourselves, and very often can use them to offset changes in the parameters of the error variable's distribution. For example, a logistic error-variable distribution with a non-zero location parameterμ (which sets the mean) is equivalent to a distribution with a zero location parameter, whereμ has been added to the intercept coefficient. Both situations produce the same value forYi* regardless of settings of explanatory variables. Similarly, an arbitrary scale parameters is equivalent to setting the scale parameter to 1 and then dividing all regression coefficients bys. In the latter case, the resulting value ofYi* will be smaller by a factor ofs than in the former case, for all sets of explanatory variables — but critically, it will always remain on the same side of 0, and hence lead to the sameYi choice.

(This predicts that the irrelevancy of the scale parameter may not carry over into more complex models where more than two choices are available.)

It turns out that this formulation is exactly equivalent to the preceding one, phrased in terms of thegeneralized linear model and without anylatent variables. This can be shown as follows, using the fact that thecumulative distribution function (CDF) of the standardlogistic distribution is thelogistic function, which is the inverse of thelogit function, i.e.

Pr(εi<x)=logit−1⁡(x){\displaystyle \Pr(\varepsilon _{i}<x)=\operatorname {logit} ^{-1}(x)}

Then:

Pr(Yi=1∣Xi)=Pr(Yi∗>0∣Xi)=Pr(β⋅Xi+εi>0)=Pr(εi>−β⋅Xi)=Pr(εi<β⋅Xi)(because the logistic distribution is symmetric)=logit−1⁡(β⋅Xi)=pi(see above){\displaystyle {\begin{aligned}\Pr(Y_{i}=1\mid \mathbf {X} _{i})&=\Pr(Y_{i}^{\ast }>0\mid \mathbf {X} _{i})\\[5pt]&=\Pr({\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon _{i}>0)\\[5pt]&=\Pr(\varepsilon _{i}>-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})\\[5pt]&=\Pr(\varepsilon _{i}<{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&&{\text{(because the logistic distribution is symmetric)}}\\[5pt]&=\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&\\[5pt]&=p_{i}&&{\text{(see above)}}\end{aligned}}}

This formulation—which is standard indiscrete choice models—makes clear the relationship between logistic regression (the "logit model") and theprobit model, which uses an error variable distributed according to a standardnormal distribution instead of a standard logistic distribution. Both the logistic and normal distributions are symmetric with a basic unimodal, "bell curve" shape. The only difference is that the logistic distribution has somewhatheavier tails, which means that it is less sensitive to outlying data (and hence somewhat morerobust to model mis-specifications or erroneous data).

Two-way latent-variable model

[edit]

Yet another formulation uses two separate latent variables:

Yi0∗=β0⋅Xi+ε0Yi1∗=β1⋅Xi+ε1{\displaystyle {\begin{aligned}Y_{i}^{0\ast }&={\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}+\varepsilon _{0}\,\\Y_{i}^{1\ast }&={\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}+\varepsilon _{1}\,\end{aligned}}}

where

ε0∼EV1⁡(0,1)ε1∼EV1⁡(0,1){\displaystyle {\begin{aligned}\varepsilon _{0}&\sim \operatorname {EV} _{1}(0,1)\\\varepsilon _{1}&\sim \operatorname {EV} _{1}(0,1)\end{aligned}}}

whereEV1(0,1) is a standard type-1extreme value distribution: i.e.

Pr(ε0=x)=Pr(ε1=x)=e−xe−e−x{\displaystyle \Pr(\varepsilon _{0}=x)=\Pr(\varepsilon _{1}=x)=e^{-x}e^{-e^{-x}}}

Then

Yi={1if Yi1∗>Yi0∗,0otherwise.{\displaystyle Y_{i}={\begin{cases}1&{\text{if }}Y_{i}^{1\ast }>Y_{i}^{0\ast },\\0&{\text{otherwise.}}\end{cases}}}

This model has a separate latent variable and a separate set of regression coefficients for each possible outcome of the dependent variable. The reason for this separation is that it makes it easy to extend logistic regression to multi-outcome categorical variables, as in themultinomial logit model. In such a model, it is natural to model each possible outcome using a different set of regression coefficients. It is also possible to motivate each of the separate latent variables as the theoreticalutility associated with making the associated choice, and thus motivate logistic regression in terms ofutility theory. (In terms of utility theory, a rational actor always chooses the choice with the greatest associated utility.) This is the approach taken by economists when formulatingdiscrete choice models, because it both provides a theoretically strong foundation and facilitates intuitions about the model, which in turn makes it easy to consider various sorts of extensions. (See the example below.)

The choice of the type-1extreme value distribution seems fairly arbitrary, but it makes the mathematics work out, and it may be possible to justify its use throughrational choice theory.

It turns out that this model is equivalent to the previous model, although this seems non-obvious, since there are now two sets of regression coefficients and error variables, and the error variables have a different distribution. In fact, this model reduces directly to the previous one with the following substitutions:

β=β1−β0{\displaystyle {\boldsymbol {\beta }}={\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0}}
ε=ε1−ε0{\displaystyle \varepsilon =\varepsilon _{1}-\varepsilon _{0}}

An intuition for this comes from the fact that, since we choose based on the maximum of two values, only their difference matters, not the exact values — and this effectively removes onedegree of freedom. Another critical fact is that the difference of two type-1 extreme-value-distributed variables is a logistic distribution, i.e.ε=ε1−ε0∼Logistic⁡(0,1).{\displaystyle \varepsilon =\varepsilon _{1}-\varepsilon _{0}\sim \operatorname {Logistic} (0,1).} We can demonstrate the equivalent as follows:

Pr(Yi=1∣Xi)=Pr(Yi1∗>Yi0∗∣Xi)=Pr(Yi1∗−Yi0∗>0∣Xi)=Pr(β1⋅Xi+ε1−(β0⋅Xi+ε0)>0)=Pr((β1⋅Xi−β0⋅Xi)+(ε1−ε0)>0)=Pr((β1−β0)⋅Xi+(ε1−ε0)>0)=Pr((β1−β0)⋅Xi+ε>0)(substitute ε as above)=Pr(β⋅Xi+ε>0)(substitute β as above)=Pr(ε>−β⋅Xi)(now, same as above model)=Pr(ε<β⋅Xi)=logit−1⁡(β⋅Xi)=pi{\displaystyle {\begin{aligned}\Pr(Y_{i}=1\mid \mathbf {X} _{i})={}&\Pr \left(Y_{i}^{1\ast }>Y_{i}^{0\ast }\mid \mathbf {X} _{i}\right)&\\[5pt]={}&\Pr \left(Y_{i}^{1\ast }-Y_{i}^{0\ast }>0\mid \mathbf {X} _{i}\right)&\\[5pt]={}&\Pr \left({\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}+\varepsilon _{1}-\left({\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}+\varepsilon _{0}\right)>0\right)&\\[5pt]={}&\Pr \left(({\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}-{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i})+(\varepsilon _{1}-\varepsilon _{0})>0\right)&\\[5pt]={}&\Pr(({\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0})\cdot \mathbf {X} _{i}+(\varepsilon _{1}-\varepsilon _{0})>0)&\\[5pt]={}&\Pr(({\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0})\cdot \mathbf {X} _{i}+\varepsilon >0)&&{\text{(substitute }}\varepsilon {\text{ as above)}}\\[5pt]={}&\Pr({\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon >0)&&{\text{(substitute }}{\boldsymbol {\beta }}{\text{ as above)}}\\[5pt]={}&\Pr(\varepsilon >-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&&{\text{(now, same as above model)}}\\[5pt]={}&\Pr(\varepsilon <{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&\\[5pt]={}&\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})\\[5pt]={}&p_{i}\end{aligned}}}

Example

[edit]
This examplepossibly containsoriginal research. Relevant discussion may be found onTalk:Logistic regression. Pleaseimprove it byverifying the claims made and addinginline citations. Statements consisting only of original research should be removed.(May 2022) (Learn how and when to remove this message)

As an example, consider a province-level election where the choice is between a right-of-center party, a left-of-center party, and a secessionist party (e.g. theParti Québécois, which wantsQuebec to secede fromCanada). We would then use three latent variables, one for each choice. Then, in accordance withutility theory, we can then interpret the latent variables as expressing theutility that results from making each of the choices. We can also interpret the regression coefficients as indicating the strength that the associated factor (i.e. explanatory variable) has in contributing to the utility — or more correctly, the amount by which a unit change in an explanatory variable changes the utility of a given choice. A voter might expect that the right-of-center party would lower taxes, especially on rich people. This would give low-income people no benefit, i.e. no change in utility (since they usually don't pay taxes); would cause moderate benefit (i.e. somewhat more money, or moderate utility increase) for middle-incoming people; would cause significant benefits for high-income people. On the other hand, the left-of-center party might be expected to raise taxes and offset it with increased welfare and other assistance for the lower and middle classes. This would cause significant positive benefit to low-income people, perhaps a weak benefit to middle-income people, and significant negative benefit to high-income people. Finally, the secessionist party would take no direct actions on the economy, but simply secede. A low-income or middle-income voter might expect basically no clear utility gain or loss from this, but a high-income voter might expect negative utility since he/she is likely to own companies, which will have a harder time doing business in such an environment and probably lose money.

These intuitions can be expressed as follows:

Estimated strength of regression coefficient for different outcomes (party choices) and different values of explanatory variables
Center-rightCenter-leftSecessionist
High-incomestrong +strong −strong −
Middle-incomemoderate +weak +none
Low-incomenonestrong +none

This clearly shows that

  1. Separate sets of regression coefficients need to exist for each choice. When phrased in terms of utility, this can be seen very easily. Different choices have different effects on net utility; furthermore, the effects vary in complex ways that depend on the characteristics of each individual, so there need to be separate sets of coefficients for each characteristic, not simply a single extra per-choice characteristic.
  2. Even though income is a continuous variable, its effect on utility is too complex for it to be treated as a single variable. Either it needs to be directly split up into ranges, or higher powers of income need to be added so thatpolynomial regression on income is effectively done.

As a "log-linear" model

[edit]

Yet another formulation combines the two-way latent variable formulation above with the original formulation higher up without latent variables, and in the process provides a link to one of the standard formulations of themultinomial logit.

Here, instead of writing thelogit of the probabilitiespi as a linear predictor, we separate the linear predictor into two, one for each of the two outcomes:

ln⁡Pr(Yi=0)=β0⋅Xi−ln⁡Zln⁡Pr(Yi=1)=β1⋅Xi−ln⁡Z{\displaystyle {\begin{aligned}\ln \Pr(Y_{i}=0)&={\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}-\ln Z\\\ln \Pr(Y_{i}=1)&={\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}-\ln Z\end{aligned}}}

Two separate sets of regression coefficients have been introduced, just as in the two-way latent variable model, and the two equations appear a form that writes thelogarithm of the associated probability as a linear predictor, with an extra term−ln⁡Z{\displaystyle -\ln Z} at the end. This term, as it turns out, serves as thenormalizing factor ensuring that the result is a distribution. This can be seen by exponentiating both sides:

Pr(Yi=0)=1Zeβ0⋅XiPr(Yi=1)=1Zeβ1⋅Xi{\displaystyle {\begin{aligned}\Pr(Y_{i}=0)&={\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}\\[5pt]\Pr(Y_{i}=1)&={\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}\end{aligned}}}

In this form it is clear that the purpose ofZ is to ensure that the resulting distribution overYi is in fact aprobability distribution, i.e. it sums to 1. This means thatZ is simply the sum of all un-normalized probabilities, and by dividing each probability byZ, the probabilities become "normalized". That is:

Z=eβ0⋅Xi+eβ1⋅Xi{\displaystyle Z=e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}

and the resulting equations are

Pr(Yi=0)=eβ0⋅Xieβ0⋅Xi+eβ1⋅XiPr(Yi=1)=eβ1⋅Xieβ0⋅Xi+eβ1⋅Xi.{\displaystyle {\begin{aligned}\Pr(Y_{i}=0)&={\frac {e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}\\[5pt]\Pr(Y_{i}=1)&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}.\end{aligned}}}

Or generally:

Pr(Yi=c)=eβc⋅Xi∑heβh⋅Xi{\displaystyle \Pr(Y_{i}=c)={\frac {e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}}{\sum _{h}e^{{\boldsymbol {\beta }}_{h}\cdot \mathbf {X} _{i}}}}}

This shows clearly how to generalize this formulation to more than two outcomes, as inmultinomial logit.This general formulation is exactly thesoftmax function as in

Pr(Yi=c)=softmax⁡(c,β0⋅Xi,β1⋅Xi,…).{\displaystyle \Pr(Y_{i}=c)=\operatorname {softmax} (c,{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i},{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i},\dots ).}

To prove that this is equivalent to the previous model, we start by recognizing the above model is overspecified, in thatPr(Yi=0){\displaystyle \Pr(Y_{i}=0)} andPr(Yi=1){\displaystyle \Pr(Y_{i}=1)} cannot be independently specified: ratherPr(Yi=0)+Pr(Yi=1)=1{\displaystyle \Pr(Y_{i}=0)+\Pr(Y_{i}=1)=1} so knowing one automatically determines the other. As a result, the model isnonidentifiable, in that multiple combinations ofβ0{\displaystyle {\boldsymbol {\beta }}_{0}} andβ1{\displaystyle {\boldsymbol {\beta }}_{1}} will produce the same probabilities for all possible explanatory variables. In fact, it can be seen that adding any constant vector to both of them will produce the same probabilities:

Pr(Yi=1)=e(β1+C)⋅Xie(β0+C)⋅Xi+e(β1+C)⋅Xi=eβ1⋅XieC⋅Xieβ0⋅XieC⋅Xi+eβ1⋅XieC⋅Xi=eC⋅Xieβ1⋅XieC⋅Xi(eβ0⋅Xi+eβ1⋅Xi)=eβ1⋅Xieβ0⋅Xi+eβ1⋅Xi.{\displaystyle {\begin{aligned}\Pr(Y_{i}=1)&={\frac {e^{({\boldsymbol {\beta }}_{1}+\mathbf {C} )\cdot \mathbf {X} _{i}}}{e^{({\boldsymbol {\beta }}_{0}+\mathbf {C} )\cdot \mathbf {X} _{i}}+e^{({\boldsymbol {\beta }}_{1}+\mathbf {C} )\cdot \mathbf {X} _{i}}}}\\[5pt]&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}}}\\[5pt]&={\frac {e^{\mathbf {C} \cdot \mathbf {X} _{i}}e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{\mathbf {C} \cdot \mathbf {X} _{i}}(e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}})}}\\[5pt]&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}.\end{aligned}}}

As a result, we can simplify matters, and restore identifiability, by picking an arbitrary value for one of the two vectors. We choose to setβ0=0.{\displaystyle {\boldsymbol {\beta }}_{0}=\mathbf {0} .} Then,

eβ0⋅Xi=e0⋅Xi=1{\displaystyle e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}=e^{\mathbf {0} \cdot \mathbf {X} _{i}}=1}

and so

Pr(Yi=1)=eβ1⋅Xi1+eβ1⋅Xi=11+e−β1⋅Xi=pi{\displaystyle \Pr(Y_{i}=1)={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}={\frac {1}{1+e^{-{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}=p_{i}}

which shows that this formulation is indeed equivalent to the previous formulation. (As in the two-way latent variable formulation, any settings whereβ=β1−β0{\displaystyle {\boldsymbol {\beta }}={\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0}} will produce equivalent results.)

Most treatments of themultinomial logit model start out either by extending the "log-linear" formulation presented here or the two-way latent variable formulation presented above, since both clearly show the way that the model could be extended to multi-way outcomes. In general, the presentation with latent variables is more common ineconometrics andpolitical science, wherediscrete choice models andutility theory reign, while the "log-linear" formulation here is more common incomputer science, e.g.machine learning andnatural language processing.

As a single-layer perceptron

[edit]

The model has an equivalent formulation

pi=11+e−(β0+β1x1,i+⋯+βkxk,i).{\displaystyle p_{i}={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x_{1,i}+\cdots +\beta _{k}x_{k,i})}}}.\,}

This functional form is commonly called a single-layerperceptron or single-layerartificial neural network. A single-layer neural network computes a continuous output instead of astep function. The derivative ofpi with respect toX = (x1, ...,xk) is computed from the general form:

y=11+e−f(X){\displaystyle y={\frac {1}{1+e^{-f(X)}}}}

wheref(X) is ananalytic function inX. With this choice, the single-layer neural network is identical to the logistic regression model. This function has a continuous derivative, which allows it to be used inbackpropagation. This function is also preferred because its derivative is easily calculated:

dydX=y(1−y)dfdX.{\displaystyle {\frac {\mathrm {d} y}{\mathrm {d} X}}=y(1-y){\frac {\mathrm {d} f}{\mathrm {d} X}}.\,}

In terms of binomial data

[edit]

A closely related model assumes that eachi is associated not with a single Bernoulli trial but withniindependent identically distributed trials, where the observationYi is the number of successes observed (the sum of the individual Bernoulli-distributed random variables), and hence follows abinomial distribution:

Yi∼Bin⁡(ni,pi), for i=1,…,n{\displaystyle Y_{i}\,\sim \operatorname {Bin} (n_{i},p_{i}),{\text{ for }}i=1,\dots ,n}

An example of this distribution is the fraction of seeds (pi) that germinate afterni are planted.

In terms ofexpected values, this model is expressed as follows:

pi=E⁡[Yini|Xi],{\displaystyle p_{i}=\operatorname {\mathbb {E} } \left[\left.{\frac {Y_{i}}{n_{i}}}\,\right|\,\mathbf {X} _{i}\right]\,,}

so that

logit⁡(E⁡[Yini|Xi])=logit⁡(pi)=ln⁡(pi1−pi)=β⋅Xi,{\displaystyle \operatorname {logit} \left(\operatorname {\mathbb {E} } \left[\left.{\frac {Y_{i}}{n_{i}}}\,\right|\,\mathbf {X} _{i}\right]\right)=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}\,,}

Or equivalently:

Pr(Yi=y∣Xi)=(niy)piy(1−pi)ni−y=(niy)(11+e−β⋅Xi)y(1−11+e−β⋅Xi)ni−y.{\displaystyle \Pr(Y_{i}=y\mid \mathbf {X} _{i})={n_{i} \choose y}p_{i}^{y}(1-p_{i})^{n_{i}-y}={n_{i} \choose y}\left({\frac {1}{1+e^{-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{y}\left(1-{\frac {1}{1+e^{-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{n_{i}-y}\,.}

This model can be fit using the same sorts of methods as the above more basic model.

Model fitting

[edit]

Maximum likelihood estimation (MLE)

[edit]

The regression coefficients are usually estimated usingmaximum likelihood estimation.[26][27] Unlike linear regression with normally distributed residuals, it is not possible to find a closed-form expression for the coefficient values that maximize the likelihood function so an iterative process must be used instead; for exampleNewton's method. This process begins with a tentative solution, revises it slightly to see if it can be improved, and repeats this revision until no more improvement is made, at which point the process is said to have converged.[26]

In some instances, the model may not reach convergence. Non-convergence of a model indicates that the coefficients are not meaningful because the iterative process was unable to find appropriate solutions. A failure to converge may occur for a number of reasons: having a large ratio of predictors to cases,multicollinearity,sparseness, or completeseparation.

  • Having a large ratio of variables to cases results in an overly conservative Wald statistic (discussed below) and can lead to non-convergence.Regularized logistic regression is specifically intended to be used in this situation.
  • Multicollinearity refers to unacceptably high correlations between predictors. As multicollinearity increases, coefficients remain unbiased but standard errors increase and the likelihood of model convergence decreases.[26] To detect multicollinearity amongst the predictors, one can conduct a linear regression analysis with the predictors of interest for the sole purpose of examining the tolerance statistic[26] used to assess whether multicollinearity is unacceptably high.
  • Sparseness in the data refers to having a large proportion of empty cells (cells with zero counts). Zero cell counts are particularly problematic with categorical predictors. With continuous predictors, the model can infer values for the zero cell counts, but this is not the case with categorical predictors. The model will not converge with zero cell counts for categorical predictors because the natural logarithm of zero is an undefined value so that the final solution to the model cannot be reached. To remedy this problem, researchers may collapse categories in a theoretically meaningful way or add a constant to all cells.[26]
  • Another numerical problem that may lead to a lack of convergence is complete separation, which refers to the instance in which the predictors perfectly predict the criterion – all cases are accurately classified and the likelihood maximized with infinite coefficients. In such instances, one should re-examine the data, as there may be some kind of error.[2][further explanation needed]
  • One can also take semi-parametric or non-parametric approaches, e.g., via local-likelihood or nonparametric quasi-likelihood methods, which avoid assumptions of a parametric form for the index function and is robust to the choice of the link function (e.g., probit or logit).[28]

Iteratively reweighted least squares (IRLS)

[edit]

Binary logistic regression (y=0{\displaystyle y=0} ory=1{\displaystyle y=1}) can, for example, be calculated usingiteratively reweighted least squares (IRLS), which is equivalent to maximizing thelog-likelihood of aBernoulli distributed process usingNewton's method. If the problem is written in vector matrix form, with parameterswT=[β0,β1,β2,…]{\displaystyle \mathbf {w} ^{T}=[\beta _{0},\beta _{1},\beta _{2},\ldots ]}, explanatory variablesx(i)=[1,x1(i),x2(i),…]T{\displaystyle \mathbf {x} (i)=[1,x_{1}(i),x_{2}(i),\ldots ]^{T}} and expected value of the Bernoulli distributionμ(i)=11+e−wTx(i){\displaystyle \mu (i)={\frac {1}{1+e^{-\mathbf {w} ^{T}\mathbf {x} (i)}}}}, the parametersw{\displaystyle \mathbf {w} } can be found using the following iterative algorithm:

wk+1=(XTSkX)−1XT(SkXwk+y−μk){\displaystyle \mathbf {w} _{k+1}=\left(\mathbf {X} ^{T}\mathbf {S} _{k}\mathbf {X} \right)^{-1}\mathbf {X} ^{T}\left(\mathbf {S} _{k}\mathbf {X} \mathbf {w} _{k}+\mathbf {y} -\mathbf {\boldsymbol {\mu }} _{k}\right)}

whereS=diag⁡(μ(i)(1−μ(i))){\displaystyle \mathbf {S} =\operatorname {diag} (\mu (i)(1-\mu (i)))} is a diagonal weighting matrix,μ=[μ(1),μ(2),…]{\displaystyle {\boldsymbol {\mu }}=[\mu (1),\mu (2),\ldots ]} the vector of expected values,

X=[1x1(1)x2(1)…1x1(2)x2(2)…⋮⋮⋮]{\displaystyle \mathbf {X} ={\begin{bmatrix}1&x_{1}(1)&x_{2}(1)&\ldots \\1&x_{1}(2)&x_{2}(2)&\ldots \\\vdots &\vdots &\vdots \end{bmatrix}}}

The regressor matrix andy(i)=[y(1),y(2),…]T{\displaystyle \mathbf {y} (i)=[y(1),y(2),\ldots ]^{T}} the vector of response variables. More details can be found in the literature.[29]

Bayesian

[edit]
Comparison oflogistic function with a scaled inverseprobit function (i.e. theCDF of thenormal distribution), comparingσ(x){\displaystyle \sigma (x)} vs.Φ(π8x){\textstyle \Phi ({\sqrt {\frac {\pi }{8}}}x)}, which makes the slopes the same at the origin. This shows theheavier tails of the logistic distribution.

In aBayesian statistics context,prior distributions are normally placed on the regression coefficients, for example in the form ofGaussian distributions. There is noconjugate prior of thelikelihood function in logistic regression. When Bayesian inference was performed analytically, this made theposterior distribution difficult to calculate except in very low dimensions. Now, though, automatic software such asOpenBUGS,JAGS,PyMC,Stan orTuring.jl allows these posteriors to be computed using simulation, so lack of conjugacy is not a concern. However, when the sample size or the number of parameters is large, full Bayesian simulation can be slow, and people often use approximate methods such asvariational Bayesian methods andexpectation propagation.

"Rule of ten"

[edit]
Main article:One in ten rule

Widely used, the "one in ten rule", states that logistic regression models give stable values for the explanatory variables if based on a minimum of about 10 events per explanatory variable (EPV); whereevent denotes the cases belonging to the less frequent category in the dependent variable. Thus a study designed to usek{\displaystyle k} explanatory variables for an event (e.g.myocardial infarction) expected to occur in a proportionp{\displaystyle p} of participants in the study will require a total of10k/p{\displaystyle 10k/p} participants. However, there is considerable debate about the reliability of this rule, which is based on simulation studies and lacks a secure theoretical underpinning.[30] According to some authors[31] the rule is overly conservative in some circumstances, with the authors stating, "If we (somewhat subjectively) regard confidence interval coverage less than 93 percent, type I error greater than 7 percent, or relative bias greater than 15 percent as problematic, our results indicate that problems are fairly frequent with 2–4 EPV, uncommon with 5–9 EPV, and still observed with 10–16 EPV. The worst instances of each problem were not severe with 5–9 EPV and usually comparable to those with 10–16 EPV".[32]

Others have found results that are not consistent with the above, using different criteria. A useful criterion is whether the fitted model will be expected to achieve the same predictive discrimination in a new sample as it appeared to achieve in the model development sample. For that criterion, 20 events per candidate variable may be required.[33] Also, one can argue that 96 observations are needed only to estimate the model's intercept precisely enough that the margin of error in predicted probabilities is ±0.1 with a 0.95 confidence level.[13]

Error and significance of fit

[edit]

Deviance and likelihood ratio test ─ a simple case

[edit]

In any fitting procedure, the addition of another fitting parameter to a model (e.g. the beta parameters in a logistic regression model) will almost always improve the ability of the model to predict the measured outcomes. This will be true even if the additional term has no predictive value, since the model will simply be "overfitting" to the noise in the data. The question arises as to whether the improvement gained by the addition of another fitting parameter is significant enough to recommend the inclusion of the additional term, or whether the improvement is simply that which may be expected from overfitting.

In short, for logistic regression, a statistic known as thedeviance is defined which is a measure of the error between the logistic model fit and the outcome data. In the limit of a large number of data points, the deviance ischi-squared distributed, which allows achi-squared test to be implemented in order to determine the significance of the explanatory variables.

Linear regression and logistic regression have many similarities. For example, in simple linear regression, a set ofK data points (xk,yk) are fitted to a proposed model function of the formy=b0+b1x{\displaystyle y=b_{0}+b_{1}x}. The fit is obtained by choosing theb parameters which minimize the sum of the squares of the residuals (the squared error term) for each data point:

ε2=∑k=1K(b0+b1xk−yk)2.{\displaystyle \varepsilon ^{2}=\sum _{k=1}^{K}(b_{0}+b_{1}x_{k}-y_{k})^{2}.}

The minimum value which constitutes the fit will be denoted byε^2{\displaystyle {\hat {\varepsilon }}^{2}}

The idea of anull model may be introduced, in which it is assumed that thex variable is of no use in predicting the yk outcomes: The data points are fitted to a null model function of the formy = b0 with a squared error term:

ε2=∑k=1K(b0−yk)2.{\displaystyle \varepsilon ^{2}=\sum _{k=1}^{K}(b_{0}-y_{k})^{2}.}

The fitting process consists of choosing a value ofb0 which minimizesε2{\displaystyle \varepsilon ^{2}} of the fit to the null model, denoted byεφ2{\displaystyle \varepsilon _{\varphi }^{2}} where theφ{\displaystyle \varphi } subscript denotes the null model. It is seen that the null model is optimized byb0=y¯{\displaystyle b_{0}={\overline {y}}} wherey¯{\displaystyle {\overline {y}}} is the mean of theyk values, and the optimizedεφ2{\displaystyle \varepsilon _{\varphi }^{2}} is:

ε^φ2=∑k=1K(y¯−yk)2{\displaystyle {\hat {\varepsilon }}_{\varphi }^{2}=\sum _{k=1}^{K}({\overline {y}}-y_{k})^{2}}

which is proportional to the square of the (uncorrected) sample standard deviation of theyk data points.

We can imagine a case where theyk data points are randomly assigned to the variousxk, and then fitted using the proposed model. Specifically, we can consider the fits of the proposed model to every permutation of theyk outcomes. It can be shown that the optimized error of any of these fits will never be less than the optimum error of the null model, and that the difference between these minimum error will follow achi-squared distribution, with degrees of freedom equal those of the proposed model minus those of the null model which, in this case, will be2−1=1{\displaystyle 2-1=1}. Using thechi-squared test, we may then estimate how many of these permuted sets ofyk will yield a minimum error less than or equal to the minimum error using the originalyk, and so we can estimate how significant an improvement is given by the inclusion of thex variable in the proposed model.

For logistic regression, the measure of goodness-of-fit is the likelihood functionL, or its logarithm, the log-likelihoodℓ. The likelihood functionL is analogous to theε2{\displaystyle \varepsilon ^{2}} in the linear regression case, except that the likelihood is maximized rather than minimized. Denote the maximized log-likelihood of the proposed model byℓ^{\displaystyle {\hat {\ell }}}.

In the case of simple binary logistic regression, the set ofK data points are fitted in a probabilistic sense to a function of the form:

p(x)=11+e−t{\displaystyle p(x)={\frac {1}{1+e^{-t}}}}

where⁠p(x){\displaystyle p(x)}⁠ is the probability thaty=1{\displaystyle y=1}. The log-odds are given by:

t=β0+β1x{\displaystyle t=\beta _{0}+\beta _{1}x}

and the log-likelihood is:

ℓ=∑k=1K(ykln⁡(p(xk))+(1−yk)ln⁡(1−p(xk))){\displaystyle \ell =\sum _{k=1}^{K}\left(y_{k}\ln(p(x_{k}))+(1-y_{k})\ln(1-p(x_{k}))\right)}

For the null model, the probability thaty=1{\displaystyle y=1} is given by:

pφ(x)=11+e−tφ{\displaystyle p_{\varphi }(x)={\frac {1}{1+e^{-t_{\varphi }}}}}

The log-odds for the null model are given by:

tφ=β0{\displaystyle t_{\varphi }=\beta _{0}}

and the log-likelihood is:

ℓφ=∑k=1K(ykln⁡(pφ)+(1−yk)ln⁡(1−pφ)){\displaystyle \ell _{\varphi }=\sum _{k=1}^{K}\left(y_{k}\ln(p_{\varphi })+(1-y_{k})\ln(1-p_{\varphi })\right)}

Since we havepφ=y¯{\displaystyle p_{\varphi }={\overline {y}}} at the maximum ofL, the maximum log-likelihood for the null model is

ℓ^φ=K(y¯ln⁡(y¯)+(1−y¯)ln⁡(1−y¯)){\displaystyle {\hat {\ell }}_{\varphi }=K(\,{\overline {y}}\ln({\overline {y}})+(1-{\overline {y}})\ln(1-{\overline {y}}))}

The optimumβ0{\displaystyle \beta _{0}} is:

β0=ln⁡(y¯1−y¯){\displaystyle \beta _{0}=\ln \left({\frac {\overline {y}}{1-{\overline {y}}}}\right)}

wherey¯{\displaystyle {\overline {y}}} is again the mean of theyk values. Again, we can conceptually consider the fit of the proposed model to every permutation of theyk and it can be shown that the maximum log-likelihood of these permutation fits will never be smaller than that of the null model:

ℓ^≥ℓ^φ{\displaystyle {\hat {\ell }}\geq {\hat {\ell }}_{\varphi }}

Also, as an analog to the error of the linear regression case, we may define thedeviance of a logistic regression fit as:

D=ln⁡(L^2L^φ2)=2(ℓ^−ℓ^φ){\displaystyle D=\ln \left({\frac {{\hat {L}}^{2}}{{\hat {L}}_{\varphi }^{2}}}\right)=2({\hat {\ell }}-{\hat {\ell }}_{\varphi })}

which will always be positive or zero. The reason for this choice is that not only is the deviance a good measure of the goodness of fit, it is also approximately chi-squared distributed, with the approximation improving as the number of data points (K) increases, becoming exactly chi-square distributed in the limit of an infinite number of data points. As in the case of linear regression, we may use this fact to estimate the probability that a random set of data points will give a better fit than the fit obtained by the proposed model, and so have an estimate how significantly the model is improved by including thexk data points in the proposed model.

For the simple model of student test scores described above, the maximum value of the log-likelihood of the null model isℓ^φ=−13.8629…{\displaystyle {\hat {\ell }}_{\varphi }=-13.8629\ldots } The maximum value of the log-likelihood for the simple model isℓ^=−8.02988…{\displaystyle {\hat {\ell }}=-8.02988\ldots } so that the deviance isD=2(ℓ^−ℓ^φ)=11.6661…{\displaystyle D=2({\hat {\ell }}-{\hat {\ell }}_{\varphi })=11.6661\ldots }

Using thechi-squared test of significance, the integral of thechi-squared distribution with one degree of freedom from 11.6661... to infinity is equal to 0.00063649...

This effectively means that about 6 out of a 10,000 fits to randomyk can be expected to have a better fit (smaller deviance) than the givenyk and so we can conclude that the inclusion of thex variable and data in the proposed model is a very significant improvement over the null model. In other words, we reject thenull hypothesis with1−D≈99.94%{\displaystyle 1-D\approx 99.94\%} confidence.

Goodness of fit summary

[edit]

Goodness of fit in linear regression models is generally measured usingR2. Since this has no direct analog in logistic regression, various methods[34]: ch.21  including the following can be used instead.

Deviance and likelihood ratio tests

[edit]

In linear regression analysis, one is concerned with partitioning variance via thesum of squares calculations – variance in the criterion is essentially divided into variance accounted for by the predictors and residual variance. In logistic regression analysis,deviance is used in lieu of a sum of squares calculations.[35] Deviance is analogous to the sum of squares calculations in linear regression[2] and is a measure of the lack of fit to the data in a logistic regression model.[35] When a "saturated" model is available (a model with a theoretically perfect fit), deviance is calculated by comparing a given model with the saturated model.[2] This computation gives thelikelihood-ratio test:[2]

D=−2ln⁡likelihood of the fitted modellikelihood of the saturated model.{\displaystyle D=-2\ln {\frac {\text{likelihood of the fitted model}}{\text{likelihood of the saturated model}}}.}

In the above equation,D represents the deviance and ln represents the natural logarithm. The log of this likelihood ratio (the ratio of the fitted model to the saturated model) will produce a negative value, hence the need for a negative sign.D can be shown to follow an approximatechi-squared distribution.[2] Smaller values indicate better fit as the fitted model deviates less from the saturated model. When assessed upon a chi-square distribution, nonsignificant chi-square values indicate very little unexplained variance and thus, good model fit. Conversely, a significant chi-square value indicates that a significant amount of the variance is unexplained.

When the saturated model is not available (a common case), deviance is calculated simply as −2·(log likelihood of the fitted model), and the reference to the saturated model's log likelihood can be removed from all that follows without harm.

Two measures of deviance are particularly important in logistic regression: null deviance and model deviance. The null deviance represents the difference between a model with only the intercept (which means "no predictors") and the saturated model. The model deviance represents the difference between a model with at least one predictor and the saturated model.[35] In this respect, the null model provides a baseline upon which to compare predictor models. Given that deviance is a measure of the difference between a given model and the saturated model, smaller values indicate better fit. Thus, to assess the contribution of a predictor or set of predictors, one can subtract the model deviance from the null deviance and assess the difference on aχs−p2,{\displaystyle \chi _{s-p}^{2},} chi-square distribution withdegrees of freedom[2] equal to the difference in the number of parameters estimated.

Let

Dnull=−2ln⁡likelihood of null modellikelihood of the saturated modelDfitted=−2ln⁡likelihood of fitted modellikelihood of the saturated model.{\displaystyle {\begin{aligned}D_{\text{null}}&=-2\ln {\frac {\text{likelihood of null model}}{\text{likelihood of the saturated model}}}\\[6pt]D_{\text{fitted}}&=-2\ln {\frac {\text{likelihood of fitted model}}{\text{likelihood of the saturated model}}}.\end{aligned}}}

Then the difference of both is:

Dnull−Dfitted=−2(ln⁡likelihood of null modellikelihood of the saturated model−ln⁡likelihood of fitted modellikelihood of the saturated model)=−2ln⁡(likelihood of null modellikelihood of the saturated model)(likelihood of fitted modellikelihood of the saturated model)=−2ln⁡likelihood of the null modellikelihood of fitted model.{\displaystyle {\begin{aligned}D_{\text{null}}-D_{\text{fitted}}&=-2\left(\ln {\frac {\text{likelihood of null model}}{\text{likelihood of the saturated model}}}-\ln {\frac {\text{likelihood of fitted model}}{\text{likelihood of the saturated model}}}\right)\\[6pt]&=-2\ln {\frac {\left({\dfrac {\text{likelihood of null model}}{\text{likelihood of the saturated model}}}\right)}{\left({\dfrac {\text{likelihood of fitted model}}{\text{likelihood of the saturated model}}}\right)}}\\[6pt]&=-2\ln {\frac {\text{likelihood of the null model}}{\text{likelihood of fitted model}}}.\end{aligned}}}

If the model deviance is significantly smaller than the null deviance then one can conclude that the predictor or set of predictors significantly improve the model's fit. This is analogous to theF-test used in linear regression analysis to assess the significance of prediction.[35]

Pseudo-R-squared

[edit]
Main article:Pseudo-R-squared

In linear regression the squared multiple correlation,R2 is used to assess goodness of fit as it represents the proportion of variance in the criterion that is explained by the predictors.[35] In logistic regression analysis, there is no agreed upon analogous measure, but there are several competing measures each with limitations.[35][36]

Four of the most commonly used indices and one less commonly used one are examined on this page:

  • Likelihood ratioR2L
  • Cox and SnellR2CS
  • NagelkerkeR2N
  • McFaddenR2McF
  • TjurR2T

Hosmer–Lemeshow test

[edit]

TheHosmer–Lemeshow test uses a test statistic that asymptotically follows aχ2{\displaystyle \chi ^{2}} distribution to assess whether or not the observed event rates match expected event rates in subgroups of the model population. This test is considered to be obsolete by some statisticians because of its dependence on arbitrary binning of predicted probabilities and relative low power.[37]

Coefficient significance

[edit]

After fitting the model, it is likely that researchers will want to examine the contribution of individual predictors. To do so, they will want to examine the regression coefficients. In linear regression, the regression coefficients represent the change in the criterion for each unit change in the predictor.[35] In logistic regression, however, the regression coefficients represent the change in the logit for each unit change in the predictor. Given that the logit is not intuitive, researchers are likely to focus on a predictor's effect on the exponential function of the regression coefficient – the odds ratio (seedefinition). In linear regression, the significance of a regression coefficient is assessed by computing at test. In logistic regression, there are several different tests designed to assess the significance of an individual predictor, most notably the likelihood ratio test and the Wald statistic.

Likelihood ratio test

[edit]

Thelikelihood-ratio test discussed above to assess model fit is also the recommended procedure to assess the contribution of individual "predictors" to a given model.[2][26][35] In the case of a single predictor model, one simply compares the deviance of the predictor model with that of the null model on a chi-square distribution with a single degree of freedom. If the predictor model has significantly smaller deviance (c.f. chi-square using the difference in degrees of freedom of the two models), then one can conclude that there is a significant association between the "predictor" and the outcome. Although some common statistical packages (e.g. SPSS) do provide likelihood ratio test statistics, without this computationally intensive test it would be more difficult to assess the contribution of individual predictors in the multiple logistic regression case.[citation needed] To assess the contribution of individual predictors one can enter the predictors hierarchically, comparing each new model with the previous to determine the contribution of each predictor.[35] There is some debate among statisticians about the appropriateness of so-called "stepwise" procedures.[weasel words] The fear is that they may not preserve nominal statistical properties and may become misleading.[38]

Wald statistic

[edit]

Alternatively, when assessing the contribution of individual predictors in a given model, one may examine the significance of theWald statistic. The Wald statistic, analogous to thet-test in linear regression, is used to assess the significance of coefficients. The Wald statistic is the ratio of the square of the regression coefficient to the square of the standard error of the coefficient and is asymptotically distributed as a chi-square distribution.[26]

Wj=βj2SEβj2{\displaystyle W_{j}={\frac {\beta _{j}^{2}}{SE_{\beta _{j}}^{2}}}}

Although several statistical packages (e.g., SPSS, SAS) report the Wald statistic to assess the contribution of individual predictors, the Wald statistic has limitations. When the regression coefficient is large, the standard error of the regression coefficient also tends to be larger increasing the probability ofType-II error. The Wald statistic also tends to be biased when data are sparse.[35]

Case-control sampling

[edit]

Suppose cases are rare. Then we might wish to sample them more frequently than their prevalence in the population. For example, suppose there is a disease that affects 1 person in 10,000 and to collect our data we need to do a complete physical. It may be too expensive to do thousands of physicals of healthy people in order to obtain data for only a few diseased individuals. Thus, we may evaluate more diseased individuals, perhaps all of the rare outcomes. This is also retrospective sampling, or equivalently it is called unbalanced data. As a rule of thumb, sampling controls at a rate of five times the number of cases will produce sufficient control data.[39]

Logistic regression is unique in that it may be estimated on unbalanced data, rather than randomly sampled data, and still yield correct coefficient estimates of the effects of each independent variable on the outcome. That is to say, if we form a logistic model from such data, if the model is correct in the general population, theβj{\displaystyle \beta _{j}} parameters are all correct except forβ0{\displaystyle \beta _{0}}. We can correctβ0{\displaystyle \beta _{0}} if we know the true prevalence as follows:[39]

β^0∗=β^0+log⁡π1−π−log⁡π~1−π~{\displaystyle {\widehat {\beta }}_{0}^{*}={\widehat {\beta }}_{0}+\log {\frac {\pi }{1-\pi }}-\log {{\tilde {\pi }} \over {1-{\tilde {\pi }}}}}

whereπ{\displaystyle \pi } is the true prevalence andπ~{\displaystyle {\tilde {\pi }}} is the prevalence in the sample.

Discussion

[edit]

Like other forms ofregression analysis, logistic regression makes use of one or more predictor variables that may be either continuous or categorical. Unlike ordinary linear regression, however, logistic regression is used for predicting dependent variables that takemembership in one of a limited number of categories (treating the dependent variable in the binomial case as the outcome of aBernoulli trial) rather than a continuous outcome. Given this difference, the assumptions of linear regression are violated. In particular, the residuals cannot be normally distributed. In addition, linear regression may make nonsensical predictions for a binary dependent variable. What is needed is a way to convert a binary variable into a continuous one that can take on any real value (negative or positive). To do that, binomial logistic regression first calculates theodds of the event happening for different levels of each independent variable, and then takes itslogarithm to create a continuous criterion as a transformed version of the dependent variable. The logarithm of the odds is thelogit of the probability, thelogit is defined as follows:logit⁡p=ln⁡p1−pfor 0<p<1.{\displaystyle \operatorname {logit} p=\ln {\frac {p}{1-p}}\quad {\text{for }}0<p<1\,.}

Although the dependent variable in logistic regression is Bernoulli, the logit is on an unrestricted scale.[2] The logit function is thelink function in this kind of generalized linear model, i.e.logit⁡E⁡(Y)=β0+β1x{\displaystyle \operatorname {logit} \operatorname {\mathcal {E}} (Y)=\beta _{0}+\beta _{1}x}

Y is the Bernoulli-distributed response variable andx is the predictor variable; theβ values are the linear parameters.

Thelogit of the probability of success is then fitted to the predictors. The predicted value of thelogit is converted back into predicted odds, via the inverse of the natural logarithm – theexponential function. Thus, although the observed dependent variable in binary logistic regression is a 0-or-1 variable, the logistic regression estimates the odds, as a continuous variable, that the dependent variable is a 'success'. In some applications, the odds are all that is needed. In others, a specific yes-or-no prediction is needed for whether the dependent variable is or is not a 'success'; this categorical prediction can be based on the computed odds of success, with predicted odds above some chosen cutoff value being translated into a prediction of success.

Machine learning and cross-entropy loss function

[edit]

In machine learning applications where logistic regression is used for binary classification, the MLE minimises thecross-entropy loss function.

Logistic regression is an importantmachine learning algorithm. The goal is to model the probability of a random variableY{\displaystyle Y} being 0 or 1 given experimental data.[40]

Consider ageneralized linear model function parameterized byθ{\displaystyle \theta },

hθ(X)=11+e−θTX=Pr(Y=1∣X;θ){\displaystyle h_{\theta }(X)={\frac {1}{1+e^{-\theta ^{T}X}}}=\Pr(Y=1\mid X;\theta )}

Therefore,

Pr(Y=0∣X;θ)=1−hθ(X){\displaystyle \Pr(Y=0\mid X;\theta )=1-h_{\theta }(X)}

and sinceY∈{0,1}{\displaystyle Y\in \{0,1\}}, we see thatPr(y∣X;θ){\displaystyle \Pr(y\mid X;\theta )} is given byPr(y∣X;θ)=hθ(X)y(1−hθ(X))(1−y).{\displaystyle \Pr(y\mid X;\theta )=h_{\theta }(X)^{y}(1-h_{\theta }(X))^{(1-y)}.} We now calculate thelikelihood function assuming that all the observations in the sample are independently Bernoulli distributed,

L(θ∣y;x)=Pr(Y∣X;θ)=∏iPr(yi∣xi;θ)=∏ihθ(xi)yi(1−hθ(xi))(1−yi){\displaystyle {\begin{aligned}L(\theta \mid y;x)&=\Pr(Y\mid X;\theta )\\&=\prod _{i}\Pr(y_{i}\mid x_{i};\theta )\\&=\prod _{i}h_{\theta }(x_{i})^{y_{i}}(1-h_{\theta }(x_{i}))^{(1-y_{i})}\end{aligned}}}

Typically, the log likelihood is maximized,

N−1log⁡L(θ∣y;x)=N−1∑i=1Nlog⁡Pr(yi∣xi;θ){\displaystyle N^{-1}\log L(\theta \mid y;x)=N^{-1}\sum _{i=1}^{N}\log \Pr(y_{i}\mid x_{i};\theta )}

which is maximized using optimization techniques such asgradient descent.

Assuming the(x,y){\displaystyle (x,y)} pairs are drawn uniformly from the underlying distribution, then in the limit of large N,

limN→+∞N−1∑i=1Nlog⁡Pr(yi∣xi;θ)=∑x∈X∑y∈YPr(X=x,Y=y)log⁡Pr(Y=y∣X=x;θ)=∑x∈X∑y∈YPr(X=x,Y=y)(−log⁡Pr(Y=y∣X=x)Pr(Y=y∣X=x;θ)+log⁡Pr(Y=y∣X=x))=−DKL(Y∥Yθ)−H(Y∣X){\displaystyle {\begin{aligned}&\lim \limits _{N\rightarrow +\infty }N^{-1}\sum _{i=1}^{N}\log \Pr(y_{i}\mid x_{i};\theta )=\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}\Pr(X=x,Y=y)\log \Pr(Y=y\mid X=x;\theta )\\[6pt]={}&\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}\Pr(X=x,Y=y)\left(-\log {\frac {\Pr(Y=y\mid X=x)}{\Pr(Y=y\mid X=x;\theta )}}+\log \Pr(Y=y\mid X=x)\right)\\[6pt]={}&-D_{\text{KL}}(Y\parallel Y_{\theta })-H(Y\mid X)\end{aligned}}}

whereH(Y∣X){\displaystyle H(Y\mid X)} is theconditional entropy andDKL{\displaystyle D_{\text{KL}}} is theKullback–Leibler divergence. This leads to the intuition that by maximizing the log-likelihood of a model, you are minimizing the KL divergence of your model from the maximal entropy distribution. Intuitively searching for the model that makes the fewest assumptions in its parameters.

Comparison with linear regression

[edit]

Logistic regression can be seen as a special case of thegeneralized linear model and thus analogous tolinear regression. The model of logistic regression, however, is based on quite different assumptions (about the relationship between the dependent and independent variables) from those of linear regression. In particular, the key differences between these two models can be seen in the following two features of logistic regression. First, the conditional distributiony∣x{\displaystyle y\mid x} is aBernoulli distribution rather than aGaussian distribution, because the dependent variable is binary. Second, the predicted values are probabilities and are therefore restricted to (0,1) through thelogistic distribution function because logistic regression predicts theprobability of particular outcomes rather than the outcomes themselves.

Alternatives

[edit]

A common alternative to the logistic model (logit model) is theprobit model, as the related names suggest. From the perspective ofgeneralized linear models, these differ in the choice oflink function: the logistic model uses thelogit function (inverse logistic function), while the probit model uses theprobit function (inverseerror function). Equivalently, in the latent variable interpretations of these two methods, the first assumes a standardlogistic distribution of errors and the second a standardnormal distribution of errors.[41] Othersigmoid functions or error distributions can be used instead.

Logistic regression is an alternative to Fisher's 1936 method,linear discriminant analysis.[42] If the assumptions of linear discriminant analysis hold, the conditioning can be reversed to produce logistic regression. The converse is not true, however, because logistic regression does not require the multivariate normal assumption of discriminant analysis.[43]

The assumption of linear predictor effects can easily be relaxed using techniques such asspline functions.[13]

History

[edit]

A detailed history of the logistic regression is given inCramer (2002). The logistic function was developed as a model ofpopulation growth and named "logistic" byPierre François Verhulst in the 1830s and 1840s, under the guidance ofAdolphe Quetelet; seeLogistic function § History for details.[44] In his earliest paper (1838), Verhulst did not specify how he fit the curves to the data.[45][46] In his more detailed paper (1845), Verhulst determined the three parameters of the model by making the curve pass through three observed points, which yielded poor predictions.[47][48]

The logistic function was independently developed in chemistry as a model ofautocatalysis (Wilhelm Ostwald, 1883).[49] An autocatalytic reaction is one in which one of the products is itself acatalyst for the same reaction, while the supply of one of the reactants is fixed. This naturally gives rise to the logistic equation for the same reason as population growth: the reaction is self-reinforcing but constrained.

The logistic function was independently rediscovered as a model of population growth in 1920 byRaymond Pearl andLowell Reed, published asPearl & Reed (1920), which led to its use in modern statistics. They were initially unaware of Verhulst's work and presumably learned about it fromL. Gustave du Pasquier, but they gave him little credit and did not adopt his terminology.[50] Verhulst's priority was acknowledged and the term "logistic" revived byUdny Yule in 1925 and has been followed since.[51] Pearl and Reed first applied the model to the population of the United States, and also initially fitted the curve by making it pass through three points; as with Verhulst, this again yielded poor results.[52]

In the 1930s, theprobit model was developed and systematized byChester Ittner Bliss, who coined the term "probit" inBliss (1934), and byJohn Gaddum inGaddum (1933), and the model fit bymaximum likelihood estimation byRonald A. Fisher inFisher (1935), as an addendum to Bliss's work. The probit model was principally used inbioassay, and had been preceded by earlier work dating to 1860; seeProbit model § History. The probit model influenced the subsequent development of the logit model and these models competed with each other.[53]

The logistic model was likely first used as an alternative to the probit model in bioassay byEdwin Bidwell Wilson and his studentJane Worcester inWilson & Worcester (1943).[54] However, the development of the logistic model as a general alternative to the probit model was principally due to the work ofJoseph Berkson over many decades, beginning inBerkson (1944), where he coined "logit", by analogy with "probit", and continuing throughBerkson (1951) and following years.[55] The logit model was initially dismissed as inferior to the probit model, but "gradually achieved an equal footing with the probit",[56] particularly between 1960 and 1970. By 1970, the logit model achieved parity with the probit model in use in statistics journals and thereafter surpassed it. This relative popularity was due to the adoption of the logit outside of bioassay, rather than displacing the probit within bioassay, and its informal use in practice; the logit's popularity is credited to the logit model's computational simplicity, mathematical properties, and generality, allowing its use in varied fields.[3]

Various refinements occurred during that time, notably byDavid Cox, as inCox (1958).[4]

The multinomial logit model was introduced independently inCox (1966) andTheil (1969), which greatly increased the scope of application and the popularity of the logit model.[57] In 1973Daniel McFadden linked the multinomial logit to the theory ofdiscrete choice, specificallyLuce's choice axiom, showing that the multinomial logit followed from the assumption ofindependence of irrelevant alternatives and interpreting odds of alternatives as relative preferences;[58] this gave a theoretical foundation for the logistic regression.[57]

Extensions

[edit]

There are large numbers of extensions:

See also

[edit]

References

[edit]
  1. ^Tolles, Juliana; Meurer, William J (2016). "Logistic Regression Relating Patient Characteristics to Outcomes".JAMA.316 (5):533–4.doi:10.1001/jama.2016.7653.ISSN 0098-7484.OCLC 6823603312.PMID 27483067.
  2. ^abcdefghijkHosmer, David W.; Lemeshow, Stanley (2000).Applied Logistic Regression (2nd ed.). Wiley.ISBN 978-0-471-35632-5.[page needed]
  3. ^abCramer 2002, p. 10–11.
  4. ^abWalker, SH; Duncan, DB (1967). "Estimation of the probability of an event as a function of several independent variables".Biometrika.54 (1/2):167–178.doi:10.2307/2333860.JSTOR 2333860.
  5. ^Cramer 2002, p. 8.
  6. ^Boyd, C. R.; Tolson, M. A.; Copes, W. S. (1987)."Evaluating trauma care: The TRISS method. Trauma Score and the Injury Severity Score".The Journal of Trauma.27 (4):370–378.doi:10.1097/00005373-198704000-00005.PMID 3106646.
  7. ^Kologlu, M.; Elker, D.; Altun, H.; Sayek, I. (2001). "Validation of MPI and PIA II in two different groups of patients with secondary peritonitis".Hepato-Gastroenterology.48 (37):147–51.PMID 11268952.
  8. ^Biondo, S.; Ramos, E.; Deiros, M.; Ragué, J. M.; De Oca, J.; Moreno, P.; Farran, L.; Jaurrieta, E. (2000). "Prognostic factors for mortality in left colonic peritonitis: A new scoring system".Journal of the American College of Surgeons.191 (6):635–42.doi:10.1016/S1072-7515(00)00758-4.PMID 11129812.
  9. ^Marshall, J. C.; Cook, D. J.; Christou, N. V.; Bernard, G. R.; Sprung, C. L.; Sibbald, W. J. (1995). "Multiple organ dysfunction score: A reliable descriptor of a complex clinical outcome".Critical Care Medicine.23 (10):1638–52.doi:10.1097/00003246-199510000-00007.PMID 7587228.
  10. ^Le Gall, J. R.; Lemeshow, S.; Saulnier, F. (1993). "A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study".JAMA.270 (24):2957–63.doi:10.1001/jama.1993.03510240069035.PMID 8254858.
  11. ^abDavid A. Freedman (2009).Statistical Models: Theory and Practice.Cambridge University Press. p. 128.
  12. ^Truett, J; Cornfield, J; Kannel, W (1967). "A multivariate analysis of the risk of coronary heart disease in Framingham".Journal of Chronic Diseases.20 (7):511–24.doi:10.1016/0021-9681(67)90082-3.PMID 6028270.
  13. ^abcHarrell, Frank E. (2015).Regression Modeling Strategies. Springer Series in Statistics (2nd ed.). New York; Springer.doi:10.1007/978-3-319-19425-7.ISBN 978-3-319-19424-0.
  14. ^M. Strano; B.M. Colosimo (2006). "Logistic regression analysis for experimental determination of forming limit diagrams".International Journal of Machine Tools and Manufacture.46 (6):673–682.doi:10.1016/j.ijmachtools.2005.07.005.
  15. ^Palei, S. K.; Das, S. K. (2009). "Logistic regression model for prediction of roof fall risks in bord and pillar workings in coal mines: An approach".Safety Science.47:88–96.doi:10.1016/j.ssci.2008.01.002.
  16. ^Berry, Michael J.A (1997).Data Mining Techniques For Marketing, Sales and Customer Support. Wiley. p. 10.
  17. ^Mesa-Arango, Rodrigo; Hasan, Samiul; Ukkusuri, Satish V.; Murray-Tuite, Pamela (February 2013)."Household-Level Model for Hurricane Evacuation Destination Type Choice Using Hurricane Ivan Data".Natural Hazards Review.14 (1):11–20.Bibcode:2013NHRev..14...11M.doi:10.1061/(ASCE)NH.1527-6996.0000083.ISSN 1527-6988.
  18. ^Wibbenmeyer, Matthew J.; Hand, Michael S.; Calkin, David E.; Venn, Tyron J.; Thompson, Matthew P. (June 2013)."Risk Preferences in Strategic Wildfire Decision Making: A Choice Experiment with U.S. Wildfire Managers".Risk Analysis.33 (6):1021–1037.Bibcode:2013RiskA..33.1021W.doi:10.1111/j.1539-6924.2012.01894.x.ISSN 0272-4332.PMID 23078036.S2CID 45282555.
  19. ^Lovreglio, Ruggiero; Borri, Dino; dell’Olio, Luigi; Ibeas, Angel (2014-02-01)."A discrete choice model based on random utilities for exit choice in emergency evacuations".Safety Science.62:418–426.doi:10.1016/j.ssci.2013.10.004.ISSN 0925-7535.
  20. ^"Logistic Regression".CORP-MIDS1 (MDS). Retrieved2024-03-16.
  21. ^Neyman, J.;Pearson, E. S. (1933),"On the problem of the most efficient tests of statistical hypotheses"(PDF),Philosophical Transactions of the Royal Society of London A,231 (694–706):289–337,Bibcode:1933RSPTA.231..289N,doi:10.1098/rsta.1933.0009,JSTOR 91247
  22. ^"How to Interpret Odds Ratio in Logistic Regression?". Institute for Digital Research and Education.
  23. ^Everitt, Brian (1998).The Cambridge Dictionary of Statistics. Cambridge, UK New York: Cambridge University Press.ISBN 978-0-521-59346-5.
  24. ^For example, the indicator function in this case could be defined asΔ(n,y)=1−(y−n)2{\displaystyle \Delta (n,y)=1-(y-n)^{2}}
  25. ^Malouf, Robert (2002)."A comparison of algorithms for maximum entropy parameter estimation".Proceedings of the Sixth Conference on Natural Language Learning (CoNLL-2002). pp. 49–55.doi:10.3115/1118853.1118871.
  26. ^abcdefgMenard, Scott W. (2002).Applied Logistic Regression (2nd ed.). SAGE.ISBN 978-0-7619-2208-7.[page needed]
  27. ^Gourieroux, Christian; Monfort, Alain (1981). "Asymptotic Properties of the Maximum Likelihood Estimator in Dichotomous Logit Models".Journal of Econometrics.17 (1):83–97.doi:10.1016/0304-4076(81)90060-9.
  28. ^Park, Byeong U.; Simar, Léopold; Zelenyuk, Valentin (2017)."Nonparametric estimation of dynamic discrete choice models for time series data"(PDF).Computational Statistics & Data Analysis.108:97–120.doi:10.1016/j.csda.2016.10.024.
  29. ^Murphy, Kevin P. (2012).Machine Learning – A Probabilistic Perspective. The MIT Press. p. 245.ISBN 978-0-262-01802-9.
  30. ^Van Smeden, M.; De Groot, J. A.; Moons, K. G.; Collins, G. S.; Altman, D. G.; Eijkemans, M. J.; Reitsma, J. B. (2016)."No rationale for 1 variable per 10 events criterion for binary logistic regression analysis".BMC Medical Research Methodology.16 (1): 163.doi:10.1186/s12874-016-0267-3.PMC 5122171.PMID 27881078.
  31. ^Peduzzi, P; Concato, J; Kemper, E; Holford, TR; Feinstein, AR (December 1996)."A simulation study of the number of events per variable in logistic regression analysis".Journal of Clinical Epidemiology.49 (12):1373–9.doi:10.1016/s0895-4356(96)00236-3.PMID 8970487.
  32. ^Vittinghoff, E.; McCulloch, C. E. (12 January 2007)."Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression".American Journal of Epidemiology.165 (6):710–718.doi:10.1093/aje/kwk052.PMID 17182981.
  33. ^van der Ploeg, Tjeerd; Austin, Peter C.; Steyerberg, Ewout W. (2014)."Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints".BMC Medical Research Methodology.14: 137.doi:10.1186/1471-2288-14-137.PMC 4289553.PMID 25532820.
  34. ^Greene, William N. (2003).Econometric Analysis (Fifth ed.). Prentice-Hall.ISBN 978-0-13-066189-0.
  35. ^abcdefghijCohen, Jacob; Cohen, Patricia; West, Steven G.;Aiken, Leona S. (2002).Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.). Routledge.ISBN 978-0-8058-2223-6.[page needed]
  36. ^Allison, Paul D."Measures of fit for logistic regression"(PDF). Statistical Horizons LLC and the University of Pennsylvania.
  37. ^Hosmer, D.W. (1997). "A comparison of goodness-of-fit tests for the logistic regression model".Stat Med.16 (9):965–980.doi:10.1002/(sici)1097-0258(19970515)16:9<965::aid-sim509>3.3.co;2-f.PMID 9160492.
  38. ^Harrell, Frank E. (2010).Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer.ISBN 978-1-4419-2918-1.[page needed]
  39. ^abhttps://class.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/classification.pdf slide 16
  40. ^Ng, Andrew (2000)."CS229 Lecture Notes"(PDF).CS229 Lecture Notes:16–19.
  41. ^Rodríguez, G. (2007).Lecture Notes on Generalized Linear Models. pp. Chapter 3, page 45.
  42. ^Gareth James; Daniela Witten; Trevor Hastie; Robert Tibshirani (2013).An Introduction to Statistical Learning. Springer. p. 6.
  43. ^Pohar, Maja; Blas, Mateja; Turk, Sandra (2004)."Comparison of Logistic Regression and Linear Discriminant Analysis: A Simulation Study".Metodološki Zvezki.1 (1).
  44. ^Cramer 2002, pp. 3–5.
  45. ^Verhulst, Pierre-François (1838)."Notice sur la loi que la population poursuit dans son accroissement"(PDF).Correspondance Mathématique et Physique.10:113–121. Retrieved3 December 2014.
  46. ^Cramer 2002, p. 4, "He did not say how he fitted the curves."
  47. ^Verhulst, Pierre-François (1845)."Recherches mathématiques sur la loi d'accroissement de la population" [Mathematical Researches into the Law of Population Growth Increase].Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles.18. Retrieved2013-02-18.
  48. ^Cramer 2002, p. 4.
  49. ^Cramer 2002, p. 7.
  50. ^Cramer 2002, p. 6.
  51. ^Cramer 2002, p. 6–7.
  52. ^Cramer 2002, p. 5.
  53. ^Cramer 2002, p. 7–9.
  54. ^Cramer 2002, p. 9.
  55. ^Cramer 2002, p. 8, "As far as I can see the introduction of the logistics as an alternative to the normal probability function is the work of a single person, Joseph Berkson (1899–1982), ..."
  56. ^Cramer 2002, p. 11.
  57. ^abCramer 2002, p. 13.
  58. ^McFadden, Daniel (1973)."Conditional Logit Analysis of Qualitative Choice Behavior"(PDF). In P. Zarembka (ed.).Frontiers in Econometrics. New York: Academic Press. pp. 105–142. Archived fromthe original(PDF) on 2018-11-27. Retrieved2019-04-20.

Sources

[edit]

External links

[edit]
Wikiversity has learning resources aboutLogistic regression
Continuous data
Center
Dispersion
Shape
Count data
Summary tables
Dependence
Graphics
Study design
Survey methodology
Controlled experiments
Adaptive designs
Observational studies
Statistical theory
Frequentist inference
Point estimation
Interval estimation
Testing hypotheses
Parametric tests
Specific tests
Goodness of fit
Rank statistics
Bayesian inference
Correlation
Regression analysis (see alsoTemplate:Least squares and regression analysis
Linear regression
Non-standard predictors
Generalized linear model
Partition of variance
Categorical
Multivariate
Time-series
General
Specific tests
Time domain
Frequency domain
Survival
Survival function
Hazard function
Test
Biostatistics
Engineering statistics
Social statistics
Spatial statistics
International
National
Other
Retrieved from "https://en.wikipedia.org/w/index.php?title=Logistic_regression&oldid=1302219996"
Categories:
Hidden categories:
[8]ページ先頭
©2009-2025 Movatter.jp