31:[] 2f:{"title":"Supervised Learning - Deep Learning - Part 3","cells":[{"id":0,"value":"md `# Supervised Learning - Deep Learning - Part 3\nSummary sheet of mathematical formulas used during a course in \"Deep Learning\"\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":28,"value":"md `## Notations\n| | |\n|---|---|\n|n| number of input features|\n|m| number of training examples|\n|(i)| refers to the i-th training example|\n|[l]| refers to the l-th layer in the network|\n|L| refers to the output layer in the network, this is also the number of hidden layers plus the output layer|\n|${tex` \\large \\{x^{(i)},y^{(i)}\\}`}| i-th traning example, x: input features, y output value (labeled value)|\n|${tex`\\large \\hat y`} or a| prediction, for the same x, we want ${tex `\\large \\hat y \\approx y`}, also noted as a (activation)|\n|g| activation function|\n|{W, b}| training model (learning parameters)|\n|${tex `\\large \\alpha`}| learning rate|\n|${tex `\\large \\lambda`}| regularization parameter|\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"md `

\n## Batch Normalization\nBatch normalization makes hyperparameter search problem much easier, makes neural network much more robust\n### Normalizing Inputs\nNormalizing Inputs to speed up learning\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"md `\n

\n${tex `\n\\Large\n \\mu = \\frac{1}{m} \\cdot \\displaystyle\\sum_{i=1}^m x^{(i)}, \\;\\;\\;\\;\n\\sigma^2 = \\frac{1}{m} \\cdot \\displaystyle\\sum_{i=1}^m (x^{(i)} - \\mu)^{2}\n\\\\\n\\;\\; \\implies X = \\frac{X - \\mu}{\\sigma}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"md `\n### Normalizing Activations\nFor a given layer l and the i-th example:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"md `\n

\n${tex `\n\\Large\n z^{[l](i)}_{\\tiny norm} \\Large = \\frac{z^{[l](i)} - \\mu^{[l]}} {\\sqrt{\\sigma^{[l]^2} + \\epsilon}} \\;\\; \\small \\text{(0 mean, unit variance)}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"md `But instead of using normalized z, we allowed the network to normalize a layer into whichever distribution is most optimal for learning by introducing 2 new learning parameters:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"md `\n

\n${tex `\n\\Large\n \\widetilde{z}^{[l](i)} \\Large = \\gamma^{[l]} \\cdot z^{[l](i)}_{\\tiny norm} + \\beta^{[l]} \\Large \\\\\n\\;\\; \\implies a^{[l](i)} = g^{[l]}( \\widetilde{z}^{[l](i)} )\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":90,"value":"md `One result of batch normalization is that we no longer need a bias vector for a batch normalized layer given that we are already shifting the normalized z\n values with the β\nparameter.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":127,"value":"md `\n### Gradient Descent\n\nNote: μ\n and σ\n are calculated on a per-batch basis while γ\n and β\n are learned parameters used across all batches.\n\n#### Forward Propagation\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":93,"value":"md `\n

\n${tex `\n\\Large\n a^{[l-1]} \\to \\left( {z}^{[l]} \\implies \\left\\langle {z}^{[l]}_{norm} \\implies \\widetilde{z}^{[l]} \\right\\rangle_{BN} \\implies g^{[l]}( \\widetilde{z}^{[l]} )\\right)_{\\tiny layer \\;l} \\to a^{[l]}\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":138,"value":"md `\n#### Backward Propagation\nThere are 2 new learning parameters γ and β, which are updated as W\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"md `\n

\n${tex `\n\\text{\\small Cost Function J:} \\\\\n\\Large\n J(\\large \\cdots, W^{[l]}, \\gamma^{[l]}, \\beta^{[l]}, \\cdots \\Large) \n\\\\\n\\text{\\small Compute gradients:}\n\\\\\n\\large\ndW^{[l]}, d\\gamma^{[l]}, d\\beta^{[l]}\n\\\\\n\n\\text{\\small Update parameters:}^{(1)} \n\\\\\nW^{[l]} = W^{[l]} - \\alpha \\cdot dW^{[l]}\\\\\n\\gamma^{[l]} = \\gamma^{[l]} - \\alpha \\cdot d\\gamma^{[l]}\\\\\n\\beta^{[l]} = \\beta^{[l]} - \\alpha \\cdot d\\beta^{[l]}\\\\\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":153,"value":"md `(1): also works with gradient descent with **momentum**, or **RMSprop**, or **Adam**`","pinned":false,"hidden":false,"mode":"ojs"},{"id":124,"value":"md `

\n## Multi Classification Problem: SOFTMAX\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":181,"value":"md `| | |\n|---|---|\n|C| number of classes = ${tex ` \\Large n^{[L]}`}|\n|${tex `c_j`}| classe j |\n\nFor the i-th example:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":189,"value":"md `\n${tex `\n\\large\n \\hat y^{(i)} \\in \\R^{C \\times 1} = \\begin{bmatrix}\n\\vdots \\\\\nc_j\\\\\n\\vdots\\\\\n\\end{bmatrix} =\n\n\\;\n\\begin{bmatrix}\n\\vdots \\\\\nP(c_j|x^{(i)})\\\\\n\\vdots\\\\\n\\end{bmatrix}\n\n\n\n`\n}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":201,"value":"md ` #### Forward Propagation, Softmax Activation Function\nFor the i-th example:\n

\n${tex `\n\\Large\n z^{[L]} = W^{[L]} \\cdot a^{[L-1]} + b^{[L]} \\\\\n t = e^{z^{[L]}} \\in \\R^{C \\times 1} \\\\\n a^{[L]} = \\hat y = \\frac{t}{ \\small \\displaystyle\\sum_{j=1}^C t_j} \\;\\; \\implies \\;\\;\n\\hat y_i = t_i / \\displaystyle\\sum_{j=1}^C t_j\n\n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":219,"value":"md ` #### Loss Function\nThis is actually the generalization of logistic regression (case with C = 2)\n

\n${tex `\n\\Large\n\\mathcal{L}( \\hat y^{(i)},y^{(i)} ) = - \\displaystyle\\sum_{j=1}^C y_j^{(i)} \\cdot log(\\hat y_j^{(i)})\n \n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":228,"value":"md ` #### Backward Propagation\n

\n${tex `\n\\Large\n J(\\large W^{[L]}, b^{[L]}, W^{[L-1]}, \\cdots \\Large) =\\frac {1} {m} \\displaystyle\\sum_{i=1}^m \\mathcal{L}(\\hat y^{(i)},y^{(i)} ) \n`}\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":237,"value":"md `\nInitiation: (justification: see https://deepnotes.io/softmax-crossentropy)\n

\n${tex `\n\\large\ndz^{[L]} = \\frac{\\partial } {\\partial z^{[L]}} \\mathcal{L}(\\cdots) = y - \\hat y = y - a^{[L]}\n\n\n`}\n

\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"md `

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":22,"value":"import {titleColor} from \"@ig2gi/supervised-learning-deep-learning-part1\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":19,"value":"import {formulaBgColor} from \"@ig2gi/supervised-learning-deep-learning-part1\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"import {formulaTxtColor} from \"@ig2gi/supervised-learning-deep-learning-part1\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}