31:[] 2f:{"title":"Supervised Learning - Deep Learning - Part 2","cells":[{"id":0,"value":"md `# Supervised Learning - Deep Learning - Part 2\nSummary sheet of mathematical formulas used during a course in \"Deep Learning\"\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":44,"value":"md `## Notations\n| | |\n|---|---|\n|n| number of input features|\n|m| number of training examples|\n|(i)| refers to the i-th training example|\n|[l]| refers to the l-th layer in the network|\n|L| refers to the output layer in the network, this is also the number of hidden layers plus the output layer|\n|${tex` \\large \\{x^{(i)},y^{(i)}\\}`}| i-th traning example, x: input features, y output value (labeled value)|\n|${tex`\\large \\hat y`} or a| prediction, for the same x, we want ${tex `\\large \\hat y \\approx y`}, also noted as a (activation)|\n|g| activation function|\n|{W, b}| training model (learning parameters)|\n|${tex `\\large \\alpha`}| learning rate|\n|${tex `\\large \\lambda`}| regularization parameter|\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"md `

\n## Regularization\nRegularization reduce overfitting.\n### L2 Regularization\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":57,"value":"md `#### Cost Function J`","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"md `\n

\n${tex `\n\\Large\n J(\\large W^{[L]}, b^{[L]}, W^{[L-1]}, \\cdots \\Large) =\\frac{1}{m} \\displaystyle\\sum_{i=1}^m \\mathcal{L}( a^{[L](i)},y^{(i)} ) + \\frac {\\lambda} {2m} \\displaystyle\\sum_{l=1}^L \\parallel W^{[l]}\\parallel^{2}_{F} \\\\\n \\;\\; \\small \\text{with} \\Large \\parallel W^{[l]}\\parallel^{2}_{F} = \\displaystyle\\sum_{i=1}^{n^{[l]}} \\displaystyle\\sum_{j=1}^{n^{[l-1]}} W^{[l]^{2}}_{ij} \\;\\;\\; \\small \\text{\"Frobenius Norm\"}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":54,"value":"md `#### Impact on gradient descent`","pinned":false,"hidden":false,"mode":"ojs"},{"id":60,"value":"md `\nSee Deep Learning - Part1 \n\n${tex `\n\\large\n\\text{given} \\;\\; dW^{[l]}_{BP} = dW^{[l]} \\text{(\\small from backward propagation \\large)}\\\\\n\\Large\ndW^{[l]} = \\frac{\\partial } {\\partial W^{[l]}} J(\\cdots) = dW^{[l]}_{BP} + \\frac{\\lambda}{m} \\cdot W^{[l]} \\\\\n`}\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":78,"value":"md `\nUpdate W parameters:\n\n${tex `\n\\large\nW^{[l]} = W^{[l]} - \\alpha dW^{[l]} = W^{[l]} - \\alpha \\cdot (dW^{[l]}_{BP} + \\frac{\\lambda}{m} \\cdot W^{[l]}) \n\n`}\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":91,"value":"md `\n

\n${tex `\n\\Large\n \\implies W^{[l]} = (1 - \\frac{\\alpha \\cdot \\lambda}{m}) W^{[l]} - \\alpha \\cdot dW^{[l]}_{BP} \\;\\;\\;\\; \\text{\\small (also known as \"weight decay\")}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":112,"value":"md `### Inverted Dropout Regularization\n Dropout randomly shuts down some neurons in hidden layers in each training iteration. The idea behind drop-out is that at each iteration, you train a different model that uses only a subset of your neurons. With dropout, your neurons thus become less sensitive to the activation of one other specific neuron, because that other neuron might be shut down at any time.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":131,"value":"md `\n${tex `\n\\large\n\nA^{[l]} \\in \\R^{n^{[l]} \\times m}\\;\\;: A^{[l]} = \n\\begin{bmatrix}\n\\mid & \\mid & & \\mid \\\\\na^{[l](1)} & a^{[l](2)} & \\ldots & a^{[l](m)} \\\\\n\\mid & \\mid & & \\mid \\\\\n\\end{bmatrix}\\\\\nD^{[l]} \\in \\R^{n^{[l]} \\times m}\\;\\;: D^{[l]} = \n\\begin{bmatrix}\n\\mid & \\mid & & \\mid \\\\\nd^{[l](1)} & d^{[l](2)} & \\ldots & d^{[l](m)} \\\\\n\\mid & \\mid & & \\mid \\\\\n\\end{bmatrix}\\\\\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":137,"value":"md `Set each entry of D[l] to be 1 with probability (keep_prob), and 0 otherwise.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":140,"value":"md `#### Forward Propagation\n

\n${tex `\n\\Large\n A^{[l]} = A^{[l]} * D^{[l]} \\;\\;\\;\\; \\text{\\small (apply mask: shutdown some neurons)} \\\\\n\\;\\\\\n\\Large\nA^{[l]} = \\frac{1}{keep\\_prob} \\cdot A^{[l]} \\;\\;\\;\\; \\text{\\small (also known as \"inverted dropout\")}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":156,"value":"md `#### Backward Propagation\nWe have to shut down the same neurons during forward propagsation, by reapplying the same mask D[l]\n to dA1.\n

\n${tex `\n\\Large\n dA^{[l]} = dA^{[l]} * D^{[l]} \\;\\;\\;\\; \\text{\\small (apply mask: shutdown some neurons)} \\\\\n\\;\\\\\n\\Large\ndA^{[l]} = \\frac{1}{keep\\_prob} \\cdot dA^{[l]} \\;\\;\\;\\; \\text{\\small (also known as \"inverted dropout\")}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":126,"value":"md `

\n## Optimizazion\n\n### 1. Mini Batch Gradient Descent\nThe difference between batch gradient descent, mini-batch gradient descent and stochastic gradient descent is the number of examples you use to perform one update step. \n\nBecause mini-batch gradient descent makes a parameter update after seeing just a subset of examples, the direction of the update has some variance, and so the path taken by mini-batch gradient descent will \"oscillate\" toward convergence. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":162,"value":"md `#### Batch Gradient Descent\nWe take gradient steps with respect to all **m**\n examples on each step.\n\n${tex `\n\n\\text{Batch size = m}\\\\\n\nX \\in \\R^{n \\times m}\\;\\;: X = \n\\begin{bmatrix}\n\\mid & \\mid & & \\mid \\\\\nx^{(1)} & x^{(2)} & \\ldots & x^{(m)} \\\\\n\\mid & \\mid & & \\mid \\\\\n\\end{bmatrix} \n\\;\\;\\;\\;\nY \\in \\R^{n^{[L]} \\times m}\\;\\;: Y = \n\\begin{bmatrix}\n\\mid & \\mid & & \\mid \\\\\ny^{(1)} & y^{(2)} & \\ldots & y^{(m)} \\\\\n\\mid & \\mid & & \\mid \\\\\n\\end{bmatrix}\\\\\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":183,"value":"md `#### Mini-Batch Gradient Descent\nMini-batch gradient descent uses an intermediate number of examples for each step. With mini-batch gradient descent, you loop over the mini-batches instead of looping over individual training examples.\n\n${tex `\n\n\\text{Batch size = s < m, \\;\\;b = batch number} \\\\\n\nX^{\\{b\\}} \\in \\R^{n \\times s}\\;\\;: X^{\\{b\\}} = \n\\begin{bmatrix}\n\\mid & \\mid & \\mid \\\\\nx^{(1)} & \\ldots & x^{(s)} \\\\\n\\mid & \\mid & \\mid \\\\\n\\end{bmatrix} \n\\;\\;\\;\\;\nY^{\\{b\\}} \\in \\R^{n^{[L]} \\times s}\\;\\;: Y^{\\{b\\}} = \n\\begin{bmatrix}\n\\mid & \\mid & \\mid \\\\\ny^{(1)} & \\ldots & y^{(s)} \\\\\n\\mid & \\mid & \\mid \\\\\n\\end{bmatrix}\\\\\n\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"md `#### Stochastic Gradient Descent\nIt is equivalent to mini-batch gradient descent where each mini-batch has just 1 example. What changes is that you would be computing gradients on just one training example at a time, rather than on the whole training set.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":228,"value":"md `### 2. Momentum: Mini Batch Gradient Descent With Momentum\nUsing momentum can reduce the oscillations introduced by mini batch grandient descent. Momentum takes into account the past gradients to smooth out the update (exponentially weighted average)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":353,"value":"md `#### Exponentially Weighted Average\n${tex `\n\\LARGE\nV_t = \\beta_1 V_{t-1} + (1 - \\beta_1) \\cdot \\theta_t\\\\ \n\\small \\;\\;\\;\\; \\text{bias correction use} \\;\\; \\Large \\frac{V_t}{1 - \\beta_1^{t}} \\\\\n\\LARGE\n\\beta_{1} = ${beta}\n\n`}\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":401,"value":"md `Example of exponentially weighted average on temperature anomalies (see https://observablehq.com/@ig2gi/global-surface-temperature-change)`","pinned":false,"hidden":false,"mode":"ojs"},{"id":321,"value":"viewof beta = slider({\n min: 0,\n max: 1,\n value: 0.8,\n description: \"beta1\"\n})\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":269,"value":"chartMomentum = {\n \n const svg = d3.select(DOM.svg(width, 200));\n xAxis(svg, 180, true);\n y2Axis(svg);\n plot (svg, \"90S-64S\", \"gray\", \"90S-64S\", \"1\", y2, true, false, false);\n \n const l = d3.line()\n .curve(d3.curveMonotoneX)\n .x(d => x(d.year))\n .y(d => y2(d.temp));\n \n const curve = svg.append(\"g\")\n .append(\"path\")\n .datum(dataMomentum)\n .attr(\"fill\", \"none\")\n .attr(\"stroke\", \"#C0392B\")\n .attr(\"stroke-linejoin\", \"round\")\n .attr(\"stroke-linecap\", \"round\")\n //.attr(\"stroke-dasharray\", dasharray ? dasharray : \"1,2\")\n .attr(\"stroke-width\", 2)\n .attr(\"d\", l);\n \n \n return svg.node();\n \n}\n \n ","pinned":false,"hidden":false,"mode":"ojs"},{"id":380,"value":"md `\nGradient Descent:\n

\n${tex `\n\n\\Large\n\\text{on iteration t \\{}\\\\\n\\;\\; \\text{\\large compute dW and db for each layer and for the current mini batch}\\\\ \n\\large\n\\;\\; \\text{\\large compute momentums}\\\\ \n\\;\\;\\;\\; V_{dW^{[l]}} = \\beta_1 \\cdot V_{dW^{[l]}} + (1 - \\beta_1) \\cdot dW^{[l]} \\\\\n\\;\\;\\;\\; V_{db^{[l]}} = \\beta_1 \\cdot V_{db^{[l]}} + (1 - \\beta_1) \\cdot db^{[l]} \\\\\n\\large\n\\\\\n\\;\\; \\text{update parameters}\\\\ \n \\;\\;\\;\\; W^{[l]} = W^{[l]} - \\alpha \\cdot V_{dW^{[l]}} \\\\ \\large\n \\;\\;\\;\\; B^{[l]} = B^{[l]} - \\alpha \\cdot V_{db^{[l]}}\n\\\\\n\\text{ \\}}\n`}\n

\n`\n\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":409,"value":"md `### 3. RMSprop: \"root mean squared prop\"\nThe weighted average is done on the squares of derivatives, dumping out the directions in which there are the most important oscillations\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":421,"value":"md `\n\n

\n${tex `\n\n\\large\n\\;\\;\\;\\; S_{dW^{[l]}} = \\beta_2 \\cdot S_{dW^{[l]}} + (1 - \\beta_2) \\cdot dW^{[l]^2} \\\\\n\\;\\;\\;\\; S_{db^{[l]}} = \\beta_2 \\cdot S_{db^{[l]}} + (1 - \\beta_2) \\cdot db^{[l]^2} \\\\\n\\large\n\\\\\n\\;\n\\\\\n \\;\\;\\;\\; W^{[l]} = W^{[l]} - \\alpha \\cdot \\frac{dW^{[l]}}{\\sqrt{S_{dW^{[l]}}}} \\\\ \\large\n\\;\\;\\;\\; B^{[l]} = B^{[l]} - \\alpha \\cdot \\frac{db^{[l]}}{\\sqrt{S_{db^{[l]}}}} \\\\\n\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":413,"value":"md `### 4. ADAM: Adaptive Moment Estimation\nCombine RMSprop and Momentum .\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":445,"value":"md `\n\n

\n${tex `\n\n\\large\n\\;\\;\\;\\; V^{corr}_{dW^{[l]}} = \\frac{V_{dW^{[l]}}}{(1- \\beta_1^t)} \\;\\;,\\;\\; \\text{idem for }V^{corr}_{db^{[l]}} \\small \\;\\;\\;\\; \\text{(bias correction)} \\\\\n\\large\n\\;\\;\\;\\; S^{corr}_{dW^{[l]}} = \\frac{S_{dW^{[l]}}}{(1- \\beta_2^t)} \\;\\;,\\;\\; \\text{idem for }S^{corr}_{db^{[l]}} \\small \\;\\;\\;\\; \\text{(bias correction)} \\\\\n\\\\\n\\;\n\\\\ \\large\n \\;\\;\\;\\; W^{[l]} = W^{[l]} - \\alpha \\cdot \\frac{V^{corr}_{dW^{[l]}} }{(\\sqrt{S^{corr}_{dW^{[l]}}} + \\epsilon)} \\\\ \\large\n\\;\\;\\;\\; B^{[l]} = B^{[l]} - \\alpha \\cdot \\frac{V^{corr}_{db^{[l]}}}{(\\sqrt{S^{corr}_{db^{[l]}}} + \\epsilon)} \\\\\n\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":472,"value":"md ` #### Hyper parameters:\n| | | |\n|---|---|---|\n|${tex ` \\large \\alpha`}| need to be tune|learning rate|\n|${tex ` \\large \\beta_1`}| 0.9|momentum (${tex ` dW`})|\n|${tex ` \\large \\beta_2`}| 0.999|RMSprop (${tex ` dW^2`})|\n|${tex ` \\large \\epsilon`}| ${tex ` 10^{-8}`}| |\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"md `

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"titleColor = \"#1F618D\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"formulaTxtColor = \"#34495E\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":10,"value":"formulaBgColor = \"#F4F6F6\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":303,"value":"dataMomentum = {\n const data = [];\n let V_prev = 0;\n let V = 0;\n let Vcorr = 0;\n d3.range(1880, 2019, 1).forEach((y, i) => {\n V = beta * V_prev + (1 - beta) * temp(y) ;\n Vcorr = V / (1 - Math.pow(beta, (i+1)));\n data.push({\n year: y,\n temp: Vcorr\n });\n V_prev = V;\n });\n return data;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":308,"value":"test = temp(1880)","pinned":false,"hidden":false,"mode":"ojs"},{"id":304,"value":"temp = (t) => data.filter(d => d.year === t)[0].temp","pinned":false,"hidden":false,"mode":"ojs"},{"id":261,"value":"data = {\n const zone = \"90S-64S\";\n \n\n return [...original.map(d => {\n return { \"year\": parseInt(d.Year),\n \"lat\":zone,\n \"temp\": parseFloat(d[zone])\n };\n })];\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":289,"value":"import {y2Axis} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"import {xAxis} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":242,"value":"import {original} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":333,"value":"import {x} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":273,"value":"import {y2} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":276,"value":"import {plot} from \"@ig2gi/global-surface-temperature-change\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":341,"value":"import {slider} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}