31:T413,

## Least squares estimation >
> The loss function can be viewed as the sum of the errors of each sample, and the error of each sample is the difference ${tex`y_i`} between ${tex`w^{T} x_{i}`}: >
> ${tex.block` L(w)=\sum_{i=1}^{N}\left\|w^{T} x_{i}-y_{i}\right\|_{2}^{2} \\ =\sum_{i=1}^{N}\left(w^{T} x_{i}-y_{i}\right)^{2} \\ =\underbrace{(w^{T} x_{1} - y_1 \quad w^{T} x_{2} - y_2 \quad \cdots \quad w^{T} x_{N} - y_N)}_{\underbrace{(w^{T} x_{1} \quad w^{T} x_{2} \quad \cdots \quad w^{T} x_{N}) - (y_1 \quad y_2 \quad \cdots \quad y_N)}_{w^{T} X^{T} - Y^{T}}} \left(\begin{array}{c} w^{T} x_{1} - y_{1} \\ w^{T} x_{2} - y_{2} \\ \vdots \\ w^{T} x_{N} - y_{N} \end{array}\right) \\~\\ =\left(w^{T} X^{T}-Y^{T}\right)(X w-Y) \\ =w^{T} X^{T} X w-w^{T} X^{T} Y-Y^{T} X w+Y^{T} Y \\ =w^{T} X^{T} X w-2 w^{T} X^{T} Y+Y^{T} Y `}
> ${tex.block` \begin{gathered} \hat{w}=\operatorname{argmin} L(w) \\ \frac{\partial L(w)}{\partial w}=2 X^{T} X w-2 X^{T} Y=0 \\ \Rightarrow w=\left(X^{T} X\right)^{-1} X^{T} Y \end{gathered} `}32:T69a,

## Maximum likelihood estimation >
> The difference between the actual value and the estimated value can be considered as a Gaussian noise (${tex`\varepsilon`}). >
>> ${tex`y=f(w)+\varepsilon=w^{T} x+\varepsilon \quad \text{ where } \varepsilon \sim N\left(0, \sigma^{2}\right) `} > ${tex.block` \\~\\ \Rightarrow y \mid x ; w \sim N\left(w^{T} x, \sigma^{2}\right) \\~\\ \Rightarrow P(y \mid x ; w)=\frac{1}{\sqrt{2 \pi} \sigma} \exp \left\{-\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}\right\} \\~\\~\\ `} >> ${tex` L(w)=\log P(Y \mid X ; w) \\ `} > ${tex.block` =\log \prod_{i=1}^{N} P\left(y_{i} \mid x_{i} ; w\right) \\ =\sum_{i=1}^{N} \log P\left(y_{i} \mid x_{i} ; w\right) \\ =\sum_{i=1}^{N}\left(\log \frac{1}{\sqrt{2 \pi}}+\log \exp \left\{-\frac{\left(y_{i}-w^{T} x_{i}\right)^{2}}{2 \sigma^{2}}\right\}\right) \\ =\sum_{i=1}^{N}\left(\log \frac{1}{\sqrt{2 \pi}}-\frac{\left(y_{i}-w^{T} x_{i}\right)^{2}}{2 \sigma^{2}}\right) \\~\\~\\ `} >> ${tex` \hat{w}=\underset{w}{\operatorname{argmax} L(w)} `} > ${tex.block` \\=\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N}\left(\log \frac{1}{\sqrt{2 \pi}}-\frac{\left(y_{i}-w^{T} x_{i}\right)^{2}}{2 \sigma^{2}}\right) \\=\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N}-\frac{\left(y_{i}-w^{T} x_{i}\right)^{2}}{2 \sigma^{2}} \\ =\underset{w}{\operatorname{argmin}} \sum_{i=1}^{N}\left(y_{i}-w^{T} x_{i}\right)^{2} \\ =\underset{w}{\operatorname{argmin}} \sum_{i=1}^{N}\left\|w^{T} x_{i}-y_{i}\right\|_{2}^{2} \\ \scriptsize {(Least \ squares \ estimation) } `} >
> Therefore, The least squares estimation is equivalent to the maximum likelihood estimation when the difference is Gaussian noise.33:T71a,

## Regularization >
> ${tex.block`\hat{w}=\left(X^{T} X\right)^{-1} X^{T} Y`} >
> When the number of samples (${tex`N`}) is much larger than the data dimension (${tex`p`}), ${tex`X^{T} X`} is invertible One the contrary, when the dimension is larger than the number of samples (high-dimensional small samples), ${tex`X^{T} X`} may not be invertible, which can cause over-fitting. >

> Solutions include ${tex` \left\{\begin{array}{l}\text { more data entires } \\~\\ \text { Feature selection/Feature Extraction } \\~\\ \text { Regularization}\end{array}\right. `} >

> * Feature selection refers to the removal of some features according to some rules to achieve dimensionality reduction > * Feature extraction methods, such as principal component analysis (PCA), also achieve dimensionality reduction > * Regularization methods refer to adding penalty terms to the loss function to avoid overfitting >

> The goal is to minimize the function ${tex`J(w)`}: >

> ${tex` J(w) = \underbrace{L(w)}_{loss} + \lambda \underbrace{P(w)}_{penalty} \\~\\ penalty \begin{cases}L 1 \text { regularization }(\text { Lasso }): & P(w)=\|w\|_{1} \\ L 2 \text { regularization }(\text { Ridge }): & P(w)=\|w\|_{2}^{2}\end{cases} `} >

> ${tex.block` J(w)=L(w)+\lambda P(w) \\ =\left(w^{T} X^{T}-Y^{T}\right)(X w-Y)+\lambda w^{T} w \\ =w^{T} X^{T} X w-2 w^{T} X^{T} Y+Y^{T} Y+\lambda w^{T} w \\ =w^{T}\left(X^{T} X+\lambda I\right) w-2 w^{T} X^{T} Y+Y^{T} Y \\~\\ \hat{w}=\underset{w}{\operatorname{argmin} J(w)} \\~\\ \frac{\partial J(w)}{\partial w}=2\left(X^{T} X+\lambda I\right) w-2 X^{T} Y=0 \\ \hat{w}=\left(X^{T} X+\lambda I\right)^{-1} X^{T} Y `} >
> Positive semidefinite matrix ${tex`X^{T} X`} plus diagonal matrix ${tex`\lambda I`} is invertible34:T911,

## Maximum a posteriori probability (MAP) estimation > Suppose the difference between the actual value and the estimated value can be considered as a Gaussian noise (${tex`\varepsilon`}).
> ${tex.block` y=f(w)+\varepsilon=w^{T} x+\varepsilon \quad \text{ where } \varepsilon \sim N\left(0, \sigma^{2}\right) \\~\\ \Rightarrow y \mid x ; w \sim N\left(w^{T} x, \sigma^{2}\right) \\~\\ \Rightarrow P(y \mid x ; w)=\frac{1}{\sqrt{2 \pi} \sigma} \exp \left\{-\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}\right\} \\~\\~\\ `} In addition, the parameters are assumed to follow a prior distribution: ${tex.block` w \sim N\left(0, \sigma_{0}^{2}\right) \Rightarrow P(w)=\frac{1}{\sqrt{2 \pi} \sigma_{0}} \exp \left\{-\frac{\|w\|_{2}^{2}}{2 \sigma_{0}^{2}}\right\} \\~\\ \text { Posterior probability: } P(w \mid Y)=\frac{P(Y \mid w) P(w)}{P(Y)} \\~\\ `}
> ${tex`\hat{w}=\underset{w}{\operatorname{argmax}} P(w \mid Y) \\`} >> ${tex` =\underset{w}{\operatorname{argmax}} \frac{P(Y \mid w) P(w)}{P(Y)} \\~\\ =\underset{w}{\operatorname{argmax}} P(Y \mid w) P(w) \\~\\ =\underset{w}{\operatorname{argmax}} \log P(Y \mid w) P(w) \\~\\ =\underset{w}{\operatorname{argmax}} \log \prod_{i=1}^{N} P\left(y_{i} \mid w\right) P(w) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N} \log P\left(y_{i} \mid w\right) P(w) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N} \log \left(\frac{1}{\sqrt{2 \pi} \sigma} \frac{1}{\sqrt{2 \pi} \sigma_{0}} \exp \left\{-\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}-\frac{\|w\|_{2}^{2}}{2 \sigma_{0}^{2}}\right\}\right) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N}\left(\log \frac{1}{\sqrt{2 \pi} \sigma} \frac{1}{\sqrt{2 \pi} \sigma_{0}}+\log \exp \left\{-\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}-\frac{\|w\|_{2}^{2}}{2 \sigma_{0}^{2}}\right\}\right) \\~\\ =\underset{w}{\operatorname{argmax}} \sum_{i=1}^{N}\left(-\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}-\frac{\|w\|_{2}^{2}}{2 \sigma_{0}^{2}}\right) \\~\\ =\underset{w}{\operatorname{argmin}} \sum_{i=1}^{N}\left(\frac{\left(y-w^{T} x\right)^{2}}{2 \sigma^{2}}+\frac{\|w\|_{2}^{2}}{2 \sigma_{0}^{2}}\right) \\~\\ =\underset{w}{\operatorname{argmin}} \sum_{i=1}^{N}(\underbrace{\left(y-w^{T} x\right)^{2}}_{L S E}+\underbrace{\frac{\sigma^{2}}{\sigma_{0}^{2}}}_{\lambda}\|w\|_{2}^{2}) `}35:[] 2f:{"title":"ML | 2 Linear Regression","cells":[{"id":0,"value":"# ML | 2 Linear Regression\n

","pinned":false,"hidden":false,"mode":"md"},{"id":530,"value":"Normal Equation with Regularization
\n${tex`\\hat{w}=\\left(X^{T} X+\\lambda I\\right)^{-1} X^{T} Y`}","pinned":false,"hidden":false,"mode":"md"},{"id":515,"value":"viewof pars = Inputs.form({\n sample: Inputs.range([10, 50], {step: 10, value: 30, label: \"Sample Size\"}),\n lambda: Inputs.range([0.001, 10], {step: 0.001, value: 2, label: \"Regularization Rate\"}),\n complex: Inputs.range([1, 5], {step: 1, value: 1, label: tex`\\text{Complexity } (x^n)`})\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":512,"value":"note2_plot1 = Plot.plot({\n marks:[\n Plot.ruleX([0]),\n Plot.ruleY([0]),\n Plot.dot(data1,{x:'x',y:'y'}),\n Plot.line(data1,{sort: 'x', x:'x',y:'yhat',stroke:\"red\"})\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"

\n> Suppose we have data:\n>

\n> ${tex`\n\n\\begin{gathered}\nD=\\left\\{\\left(x_{1}, y_{1}\\right),\\left(x_{2}, y_{2}\\right), \\cdots,\\left(x_{N}, y_{N}\\right)\\right\\} \\\\\nx_{i} \\in \\mathbb{R}^{p}, y_{i} \\in \\mathbb{R}, i=1,2, \\cdots, N \\\\~\\\\\nX=\\left(x_{1}, x_{2}, \\cdots, x_{N}\\right)^{T}=\\left(\\begin{array}{c}\nx_{1}^{T} \\\\\nx_{2}^{T} \\\\\n\\vdots \\\\\nx_{N}^{T}\n\\end{array}\\right)=\\left(\\begin{array}{cccc}\nx_{11} & x_{12} & \\cdots & x_{1 p} \\\\\nx_{21} & x_{22} & \\cdots & x_{2 p} \\\\\n\\vdots & \\vdots & \\ddots & \\vdots \\\\\nx_{N 1} & x_{N 2} & \\cdots & x_{N p}\n\\end{array}\\right)_{N \\times p} \\\\~\\\\\nY=\\left(\\begin{array}{c}\ny_{1} \\\\\ny_{2} \\\\\n\\vdots \\\\\ny_{N}\n\\end{array}\\right)_{N \\times 1}\n\\end{gathered}\n\n`}","pinned":false,"hidden":false,"mode":"md"},{"id":19,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":75,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":116,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":191,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":234,"value":"> The regularized least squares estimation is equivalent to the maximum a posterior estimation when the noise and the prior are Gaussian distributed.","pinned":false,"hidden":false,"mode":"md"},{"id":244,"value":"

\n> ${tex.block`\nL S E \\Leftrightarrow M L E \\ (\\text { noise is Gaussian Distributed }) \\\\\n\\text {Regularized } L S E \\Leftrightarrow M A P \\ (\\text { noise and prior are Gaussian Distributed })\n`}","pinned":false,"hidden":false,"mode":"md"},{"id":257,"value":"

\n## References\n\n> * https://youtu.be/i_Sq7qRzXiI\n> * https://youtu.be/2mUxdAhhgPY\n> * https://blog.csdn.net/qq_41485273/article/details/111770798","pinned":false,"hidden":false,"mode":"md"},{"id":261,"value":"

\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":382,"value":"data1 = aq.table({\n x: NormalEquationResult[0],\n y: NormalEquationResult[1],\n yhat: NormalEquationResult[2]\n}).objects()","pinned":false,"hidden":false,"mode":"ojs"},{"id":446,"value":"NormalEquationResult \n= py`${funcNormalEquation}(${pars.sample}, ${pars.lambda}, ${pars.complex})`","pinned":false,"hidden":false,"mode":"ojs"},{"id":436,"value":"funcNormalEquation = py`import numpy as np\nfrom sklearn.datasets import make_regression\n\ndef fne(sample = 30, lamda = 1, complex = 3):\n x,y=make_regression(n_samples=sample,n_features=1,n_informative=1,noise = 10,random_state=2)\n\n tmp = [np.ones(len(x))]\n for i in range(1,complex+1):\n tmp.append(np.power(x.flatten(), i))\n\n x_new = np.array(tmp).T \n # x_new=np.array([np.ones(len(x)),x.flatten(), np.power(x.flatten(), 2), np.power(x.flatten(), 3)]).T\n Y = (y[:, np.newaxis])\n one = np.identity(x_new.shape[1])\n one[0,0] = 0\n \n theta=np.linalg.inv(x_new.T.dot(x_new) + lamda*one).dot(x_new.T).dot(Y)\n \n yhat=x_new.dot(theta)\n \n return [x.flatten(), y, yhat.flatten(), theta.flatten(), one]\nfne`","pinned":false,"hidden":false,"mode":"ojs"},{"id":307,"value":"py = async (strings, ...expressions) => {\n let globals = {};\n const code = strings.reduce((result, string, index) => {\n if (expressions[index]) {\n const name = `x${index}`;\n globals[name] = expressions[index];\n return result + string + name;\n }\n return result + string;\n }, '');\n await pyodide.loadPackagesFromImports(code);\n const result = await pyodide.pyodide_py.eval_code_async(\n code,\n pyodide.toPy(globals)\n );\n if (result?.toJs) return result.toJs();\n return result;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":310,"value":"pyodide = {\n const pyodide = await require(\"//cdn.jsdelivr.net/pyodide/v0.19.1/full/pyodide.js\");\n return pyodide.loadPyodide({\n indexURL: \"https://cdn.jsdelivr.net/pyodide/v0.19.1/full/\"\n });\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":376,"value":"import { aq, op } from '@uwdata/arquero'","pinned":false,"hidden":false,"mode":"ojs"},{"id":265,"value":"import {Plot, d3} from \"@observablehq/plot\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W35","stdlib":"1","titling":"auto"}