$\begingroup$$\endgroup$
5I've recently been exposed to the geometric intuition regarding Least Squares (OLS) regression:
The vector of the outcome variable$Y$, is not not in the linear span of$X_1, X_2, ..., X_{p-1}$:
The orthogonal vector$Y-X\hat{B}$ is that which minimizes the$l^2$-norm of the error vector and solves the least squares problem.
My question is: for the least absolute deviations (LAD) problem, where we instead want to minimize the$l^1$-norm, is there any intuition for what projection solves the LAD problem? It is not necessarily the orthogonal vector above (and LAD solutions are not necessarily unique).
This is really more of a linear algebra question than anything: What will the projection look like if we want to minimize the$l^1$-norm?
- 2$\begingroup$There is no projection associated with the $L^1$ norm. Thereis a geometric explanation, given in 1757 by R. Boscovitch, as explained (and illustrated) by Stephen Stigler in hisHistory of Statistics.$\endgroup$2024-12-02 21:23:10 +00:00CommentedDec 2, 2024 at 21:23
- $\begingroup$@whuber How can there be no projection associated with $L^1$ norm? Don't we need a $\hat{Y}$ vector in the span of $X$ to obtain coefficients (regardless of the objective function)? Thank you for pointing me to that text.$\endgroup$Liam– Liam2024-12-02 21:58:12 +00:00CommentedDec 2, 2024 at 21:58
- 1$\begingroup$That's might be some kind ofsolution but it won't enjoy the defining properties of a projection (namely, an idempotent endomorphism of a vector space).$\endgroup$2024-12-02 22:34:57 +00:00CommentedDec 2, 2024 at 22:34
- $\begingroup$@whuber Interesting. Is it fair to say that this picture breaks down when we move to an objective function in taxicab geometry? Thank you, again for the help.$\endgroup$Liam– Liam2024-12-02 23:08:12 +00:00CommentedDec 2, 2024 at 23:08
- $\begingroup$Yes, that's fair.$\endgroup$2024-12-03 15:06:08 +00:00CommentedDec 3, 2024 at 15:06