31:[] 2f:{"title":"Machine Learning in JS with TensorFlow.js (Part I)","cells":[{"id":0,"value":"md`# Machine Learning in JS with TensorFlow.js (Part I)\n\nCreated for AI Day events in Bangkok (Apr 21, 2018) and Singapore (Apr 29, 2018), this tutorial is based on:\n- [TensorFlow.js](https://js.tensorflow.org/)\n- [Introduction to the TensorFlow.js core API](https://beta.observablehq.com/@nsthorat/introduction-to-deeplearn-js)\n- [Machine Learning in JavaScript (TensorFlow Dev Summit 2018)](https://www.youtube.com/watch?v=YB-kfeNIPCE)\n\n**TensorFlow.js** is a general purpose, WebGL-accelerated numerics platform for JavaScript. It brings highly performant machine learning building blocks to your fingertips, allowing you to train neural networks in a browser or run pre-trained models in inference mode.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":121,"value":"md`## Getting Started\n\nFirst, you'd need to import \\`@tensorflow/tfjs\\``","pinned":false,"hidden":false,"mode":"ojs"},{"id":51,"value":"tf = require('@tensorflow/tfjs')","pinned":true,"hidden":false,"mode":"ojs"},{"id":56,"value":"md`Now, let's explore Tensors and basic operations in TensorFlow.js Core API by generating some synthetic data and a Mean Squared Error (MSE) loss function to be used later in this notebook.\n\n### Generate Synthetic Data\nWe will generate data points from a polynomial function with some noises.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":624,"value":"tex`y = ax^3 + bx^2 + cx + d + \\mathcal{N}(0, \\sigma^2)`","pinned":false,"hidden":false,"mode":"ojs"},{"id":183,"value":"viewof coeff = {\n const form = html`\n `;\n \n const labels = form.querySelectorAll('div span.value');\n form.value = [form.a, form.b, form.c, form.d].map(i => i.valueAsNumber);\n [form.a, form.b, form.c, form.d].forEach((d, i) => {\n d.oninput = () => {\n form.value[i] = d.valueAsNumber;\n labels[i].innerHTML = d.valueAsNumber;\n mutable isInputUpdated_a = false;\n mutable isInputUpdated_b = false;\n };\n });\n \n return form;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":213,"value":"viewof sd = {\n const form = html`\n `;\n \n const label = form.querySelector('span.value');\n form.value = form.sigma.valueAsNumber;\n form.sigma.oninput = () => {\n form.value = form.sigma.valueAsNumber;\n label.innerHTML = form.sigma.valueAsNumber;\n mutable isInputUpdated_a = false;\n mutable isInputUpdated_b = false;\n };\n \n return form; \n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":262,"value":"plot(inputData.x, inputData.y)","pinned":false,"hidden":false,"mode":"ojs"},{"id":253,"value":"inputData = {\n // tf.tidy will clean up all the GPU memory used by tensors \n // inside this function, other than the tensor that is returned.\n const data = tf.tidy(() => {\n const NUM_POINTS = 200;\n const [a, b, c, d] = coeff.map(i => tf.scalar(i)); // a = tf.scalar(coeff[0]), and so on\n\n const x = tf.randomUniform([NUM_POINTS], -1, 1);\n const y = a.mul(x.pow(tf.scalar(3)))\n .add(b.mul(x.square()))\n .add(c.mul(x))\n .add(d)\n .add(tf.randomNormal([NUM_POINTS], 0, sd));\n\n return {x, y}; \n });\n yield data;\n \n // This cell is re-evaluated and creates new tensors as we interact with the sliders to adjust parameters.\n // We need to manually dispose them to avoid memory leak. \n // See: https://beta.observablehq.com/@nsthorat/clean-up-deeplearn-js-tensor-generator-cells\n try {\n yield invalidation;\n } finally {\n data.x.dispose();\n data.y.dispose();\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":660,"value":"md`### MSE Loss\nWe will use MSE to measure pearformance of our models.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"// Loss: Mean Squared Error (MSE)\nloss = (predictions, labels) => {\n return tf.tidy(() => \n predictions.sub(labels)\n .square()\n .mean()\n );\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":136,"value":"md`## Training models in your browser\n\nThere are two approaches to training your models -- using Core API (low level) or Layer API (high level).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":333,"value":"md`### a) Training models with Core API\nIn this example, assume that we know the input data is generated from a cubic function. And we will try to estimate the values of the coefficients *a*, *b*, *c*, and *d*.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":675,"value":"mutable isInputUpdated_a = false","pinned":false,"hidden":false,"mode":"ojs"},{"id":678,"value":"{\n const button = html` `;\n const div = isInputUpdated_a ? \n html`

The input data has been updated.

` :\n html`

The input data has been changed. Click to update the input and retrain the model.

` ;\n button.disabled = isInputUpdated_a; \n button.onclick = () => {\n mutable inputData_a = inputData;\n mutable isInputUpdated_a = true; \n button.disabled = true; \n }\n return html`${button}${div}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":670,"value":"mutable inputData_a = null","pinned":false,"hidden":false,"mode":"ojs"},{"id":390,"value":"inputData_a ?\n plot(inputData_a.x, inputData_a.y, cubicPredictions) :\n html`\n

Update input to get started!

\n `","pinned":false,"hidden":false,"mode":"ojs"},{"id":777,"value":"loss_a = {\n if (!inputData_a) return null;\n const mseTensor = loss(cubicPredictions, inputData_a.y);\n const mse = mseTensor.dataSync()[0];\n mseTensor.dispose();\n return md`Mean Squared Error: ${mse}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":774,"value":"md`Adjust the number of iterations and see how it affects the predictions.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":348,"value":"viewof numIterationsParam = {\n const form = html`\n `;\n \n const label = form.querySelector('span.value');\n form.value = form.iters.valueAsNumber;\n form.iters.oninput = () => {\n form.value = form.iters.valueAsNumber;\n label.innerHTML = form.iters.valueAsNumber;\n };\n \n return form; \n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":720,"value":"{\n const isUpdated = numIterations == numIterationsParam; \n const button = html` `;\n const div = isUpdated ? \n html`

The model parameters have been updated.

` :\n html`

The model parameters have been changed. Click to retrain the model.

` ;\n button.disabled = isUpdated; \n button.onclick = () => {\n mutable numIterations = numIterationsParam;\n button.disabled = true; \n }\n return html`${button}${div}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":724,"value":"mutable numIterations = 50","pinned":false,"hidden":false,"mode":"ojs"},{"id":346,"value":"cubicPredictions = {\n if (!inputData_a) return html``;\n \n const predictions = tf.tidy(() => {\n // Model parameters: \n const a = tf.variable(tf.zeros([1]));\n const b = tf.variable(tf.zeros([1]));\n const c = tf.variable(tf.zeros([1]));\n const d = tf.variable(tf.zeros([1]));\n\n // Model: f(x) = a * x^3 + b * x^2 + c * x + d\n const f = (x) => \n a.mul(x.pow(tf.scalar(3)))\n .add(b.mul(x.square()))\n .add(c.mul(x))\n .add(d);\n\n // Optimizer: Stochastic Gradient Descent (SGD)\n const optimizer = tf.train.sgd(0.8);\n\n // Training Loop\n for (let i = 0; i < numIterations; i++) {\n optimizer.minimize(() => loss(f(inputData_a.x), inputData_a.y)); \n }\n\n const predTensor = f(inputData_a.x); \n \n // tf.tidy() will not clean up variables\n [a, b, c, d].forEach(x => { \n x.dispose(); \n });\n\n return predTensor; \n });\n yield predictions; \n \n try {\n yield invalidation;\n } finally {\n predictions.dispose();\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":404,"value":"md`### b) Training models with Layer API\nIn this example, we will train a neural network with one hidden layer. From the [Universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem), a simple neural network with a single hidden layer is sufficient to represent any function. Let's see how it works for our synthetic dataset. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":734,"value":"mutable isInputUpdated_b = false","pinned":false,"hidden":false,"mode":"ojs"},{"id":782,"value":"{\n const button = html` `;\n const div = isInputUpdated_b ? \n html`

The input data has been updated.

` :\n html`

The input data has been changed. Click to update the input and retrain the model.

` ;\n button.disabled = isInputUpdated_b; \n button.onclick = () => {\n mutable inputData_b = inputData;\n mutable isInputUpdated_b = true; \n button.disabled = true; \n }\n return html`${button}${div}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":784,"value":"mutable inputData_b = null","pinned":false,"hidden":false,"mode":"ojs"},{"id":830,"value":"inputData_b ?\n plot(inputData_b.x, inputData_b.y, nnPredictions) :\n html`\n

Update input data to get started

\n `","pinned":false,"hidden":false,"mode":"ojs"},{"id":923,"value":"loss_b = {\n if (!inputData_b) return null;\n const mseTensor = loss(nnPredictions.squeeze(), inputData.y);\n const mse = mseTensor.dataSync()[0];\n mseTensor.dispose();\n return md`Mean Squared Error: ${mse}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":786,"value":"md`Adjust the number of nodes in the hidden layer and see how it affects the predictions`","pinned":false,"hidden":false,"mode":"ojs"},{"id":805,"value":"viewof numNodesParam = {\n const form = html`\n `;\n \n const label = form.querySelector('span.value');\n form.value = form.nodes.valueAsNumber;\n form.nodes.oninput = () => {\n form.value = form.nodes.valueAsNumber;\n label.innerHTML = form.nodes.valueAsNumber;\n };\n \n return form; \n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":798,"value":"{\n const isUpdated = (numNodes == numNodesParam); \n const button = html` `;\n const div = isUpdated ? \n html`

The model parameters have been updated.

` :\n html`

The model parameters have been changed. Click to retrain the model.

` ;\n button.disabled = isUpdated; \n button.onclick = () => {\n mutable numNodes = numNodesParam;\n button.disabled = true; \n }\n return html`${button}${div}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":817,"value":"mutable numNodes = 8","pinned":false,"hidden":false,"mode":"ojs"},{"id":832,"value":"nn = {\n const model = tf.sequential({\n name: 'nn',\n layers: [\n // Neural network with one hidden layer\n tf.layers.dense({units: numNodes, activation: 'relu', inputShape: [1]}),\n tf.layers.dense({units: 1})\n ]\n });\n \n model.compile({loss: 'meanSquaredError', optimizer: 'adam'});\n return model;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":904,"value":"nnTrainInfo = {\n if (!inputData_b) return null;\n \n const TRAIN_BATCHES = 50;\n const BATCH_SIZE = 10; \n \n for (let i = 0; i < TRAIN_BATCHES; i++) {\n const history = await nn.fit(\n inputData_b.x,\n inputData_b.y,\n {batchSize: BATCH_SIZE, epochs: 1}\n );\n const loss = history.history.loss[0];\n yield {i, loss};\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":523,"value":"nnPredictions = {\n if (!inputData_b) return null;\n \n nnTrainInfo; // Just to subscribe changes in training info\n const predictions = tf.tidy(() => \n nn.model.predict(inputData_b.x.expandDims(1))\n );\n yield predictions;\n \n try {\n yield invalidation;\n } finally {\n predictions.dispose();\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1095,"value":"md`If you want to try something more advanced, like training a Convolutional Neural Network (CNN) to recognize handwritten digits, check out this [example](https://js.tensorflow.org/tutorials/mnist.html) ([code](https://github.com/tensorflow/tfjs-examples/tree/master/mnist), [demo](https://storage.googleapis.com/tfjs-examples/mnist/dist/index.html))`","pinned":false,"hidden":false,"mode":"ojs"},{"id":972,"value":"// TO-DO: Fix memory leak\nnumTensors = tf.memory().numTensors","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"vegalite = require('@observablehq/vega-lite')","pinned":false,"hidden":false,"mode":"ojs"},{"id":668,"value":"plot = async (xTensor, yTensor, predTensor) => {\n const x = await xTensor.data();\n const y = await yTensor.data(); \n const pred = predTensor ? await predTensor.data() : null; \n \n const values = [...x].map((xVal, i) => { // Spread operator: [...x] = Array.from(x)\n return predTensor ? \n {\"x\": xVal, \"y\": y[i], \"pred\": pred[i]} :\n {\"x\": xVal, \"y\": y[i]};\n });\n \n return vegalite({\n width: 400,\n height: 300,\n \"data\": { values },\n \"layer\": [{\n \"mark\": \"circle\",\n \"encoding\": {\n \"x\": {\"field\": \"x\", \"type\": \"quantitative\"},\n \"y\": {\"field\": \"y\", \"type\": \"quantitative\"}\n }\n }, {\n \"mark\": \"line\",\n \"encoding\": {\n \"x\": {\"field\": \"x\", \"type\": \"quantitative\"},\n \"y\": {\"field\": \"pred\", \"type\": \"quantitative\"} \n }\n }],\n \"config\": {\n \"circle\": {\"color\": \"#1E4865\", \"size\": 50, \"opacity\": 0.6},\n \"line\": {\"color\": \"#F1AD46\", \"strokeWidth\": 4, \"opacity\": 0.6}\n }\n })\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":685,"value":"html`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1098,"value":"md`In the next [notebook](https://beta.observablehq.com/@tvirot/machine-learning-in-javascript-with-tensorflow-js-part-ii), we will discuss how ones can load pre-trained models for inference in browser.\n\nFollow us on [Facebook](//www.facebook.com/skooldio) and [Medium](//www.medium.com/skooldio).\n\n![skooldio](https://cdn-images-1.medium.com/max/2000/1*JjMVAbfDVuVdN7DoGP5X-g@2x.png)`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}