30:T46c,html`

${[10, 8, 1].map(i => { const c = canvas(cardWhiteBalanced[i], 'width: 25%'), ctx = c.getContext('2d') ctx.strokeStyle = 'black' ctx.lineWidth = 2 for (const contour of cardShapeContours[i]) { ctx.strokeStyle = 'black' ctx.beginPath() for (const [x, y] of contour) ctx.lineTo(x, y) ctx.closePath() ctx.stroke() const xExtent = d3.extent(contour, p => p[0]), yExtent = d3.extent(contour, p => p[1]), dx = Math.floor((xExtent[1] - xExtent[0]) / 3), dy = Math.floor((yExtent[1] - yExtent[0]) / 4) ctx.strokeStyle = 'tomato' ctx.beginPath() ctx.lineTo(xExtent[0], yExtent[0]) ctx.lineTo(xExtent[1], yExtent[0]) ctx.lineTo(xExtent[1], yExtent[1]) ctx.lineTo(xExtent[0], yExtent[1]) ctx.closePath() ctx.stroke() ctx.beginPath() ctx.lineTo(xExtent[0] + dx, yExtent[0] + dy) ctx.lineTo(xExtent[1] - dx, yExtent[0] + dy) ctx.lineTo(xExtent[1] - dx, yExtent[1] - dy) ctx.lineTo(xExtent[0] + dx, yExtent[1] - dy) ctx.closePath() ctx.stroke() } return c })}

`31:[] 2e:{"title":"Cheat at Set","cells":[{"id":596,"value":"md`\n# Cheat at Set\n\nThis notebook can be read blog-like or viewed presentation-like using ← → keys. Phone runtime is at [set.visnup.com](https://set.visnup.com) and [github.com/visnup/cheat-at-set](https://github.com/visnup/cheat-at-set).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"slide.md`\n# How to cheat at life with JavaScript, computer vision, and augmented reality\n\nJSConf Hawai‘i – February 7, 2019 \n[Watch the video](https://youtu.be/5vU6shMFiPA)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1551,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":133,"value":"slide.md`\n# 👋🏼 Hello\n\nYou probably should know three things about me:\n\n1. I like games, all types of games\n2. I also program\n3. I like mischief\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":151,"value":"notes.md`\nI like lotsa games: card games 🃏, board games ♟, video games 👾, games with zombies 🧟‍♀️, games with trains 🚂, games with victory points 🏅, games with 🎲. I guess it turns out I like any game with an emoji. Weird.\n\nI also like computers: computers with screens 🖥, computers with actuators 🤖, computers with klaxxons 🚨, computers with internet ☁️. I first mistakenly formatted my hard drive (trying to install Sim Earth 🌏) in 1990 👴🏼.\n\nAnd finally, I like \"creatively solving\" things. Some people call this hacking, or mischief, or pranks. The enterprisely word for it would be \"innovation,\" but to me it's simply mischeviousness. That's also why I named my son Loki, after the Norse god of mischief. 👹\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":313,"value":"slide.md`\n $\"A$ \n\n# 👮🏻‍♀️ How to play\n\nSET® is a card-matching game. Cards have 4 features, each with 3 variations:\n\n- Color: red, purple, green\n- Number: 1, 2, 3\n- Shading: solid, striped, outline\n- Shape: diamond, oval, squiggle\n\nA set is 3 cards for which each feature is either common across all 3 cards or is different on each card.\n\nSource: [The New York Times][1]\n\n[1]: https://www.nytimes.com/puzzles/set\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":307,"value":"notes.md`\nSo, the other day I was playing this game with my friend Susan. And I'd compare this game with her to jumping into something like Fortnite or Counter Strike for the first time: you login, pick a character (choose their hair color and maybe give them a cool, rugged eye patch 'cuz they've seen some hard times and all), parachute or spawn and you have just enough time to appreciate the specular shading when _BAM!_ you've been killed by a 12 year old who's already somehow looted some crazy weapon from who knows where. And then you spawn again near this cabin or something and _BAM!_ dead again from someone who was standing behind you, camping the spawn point. Honestly, I don't even think you spawn again in Fortnite, but the point is: watch this video of playing with Susan.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":182,"value":"slide.html('bleed')`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1772,"value":"slide.md`\n# 👨‍👨‍👧 [Three-act][1] questionnaire\n\nQ: How is my protagonist dissatisfied with their life?
\nA:\n\nQ: What would it take for my protagonist to find satisfaction? (This is their goal).
\nA:\n\n[1]: https://en.wikipedia.org/wiki/Three-act_structure\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1904,"value":"slide.md`\n# 👨‍👨‍👧 [Three-act][1] questionnaire\n\nQ: How is my protagonist dissatisfied with their life?
\nA: _Can't beat Susan at Set_\n\nQ: What would it take for my protagonist to find satisfaction? (This is their goal).
\nA:\n\n[1]: https://en.wikipedia.org/wiki/Three-act_structure\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1898,"value":"slide.md`\n# 👨‍👨‍👧 [Three-act][1] questionnaire\n\nQ: How is my protagonist dissatisfied with their life?
\nA: _Can't beat Susan at Set_\n\nQ: What would it take for my protagonist to find satisfaction? (This is their goal).
\nA: _Beat Susan at Set_\n\n[1]: https://en.wikipedia.org/wiki/Three-act_structure\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1919,"value":"slide.md`\nQ: What are my protagonist’s biggest fears and character flaws?
\nA: _Probably [three things][2]..._\n\n1. \n2. \n3. \n\n[2]: http://wiki.c2.com/?LazinessImpatienceHubris\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1932,"value":"slide.md`\nQ: What are my protagonist’s biggest fears and character flaws?
\nA: _Probably [three things][2]..._\n\n1. _Laziness: Don't want to put the effort to train my brain_\n2. \n3. \n\n[2]: http://wiki.c2.com/?LazinessImpatienceHubris\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1926,"value":"slide.md`\nQ: What are my protagonist’s biggest fears and character flaws?
\nA: _Probably [three things][2]..._\n\n1. _Laziness: Don't want to put the effort to train my brain_\n2. _Impatience: Don't want to take the time to get good at this game_\n3. \n\n[2]: http://wiki.c2.com/?LazinessImpatienceHubris\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1930,"value":"slide.md`\nQ: What are my protagonist’s biggest fears and character flaws?
\nA: _Probably [three things][2]..._\n\n1. _Laziness: Don't want to put the effort to train my brain_\n2. _Impatience: Don't want to take the time to get good at this game_\n3. _Hubris: I can solve this with a computer, probably with a regex_\n\n[2]: http://wiki.c2.com/?LazinessImpatienceHubris\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":219,"value":"slide.html` $\"COMPUTER$ `","pinned":false,"hidden":false,"mode":"ojs"},{"id":62,"value":"slide.md`\n# 🗺 The plan\n\n- Camera → video → canvas\n- Identify cards\n- Extract each card and fix perspective\n- For each card, identify:\n - Color\n - Number\n - Shading\n - Shape\n- Brute force solve for sets\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":235,"value":"slide.md('small')`# Test image 🔜 <canvas>`","pinned":false,"hidden":false,"mode":"ojs"},{"id":518,"value":"viewof source = select({\n title: 'source',\n options: [\n { value: 'https://i.imgur.com/YHqc84a.jpg', label: 'Easy 1' },\n { value: 'https://i.imgur.com/eng0Vak.jpg', label: 'Easy 2' },\n // { value: 'https://i.imgur.com/ZQYFioA.jpg', label: 'Easy 3' },\n { value: 'https://i.imgur.com/1T78BGV.jpg', label: 'Easy 3' },\n { value: 'https://i.imgur.com/iUO4fUf.jpg', label: 'Vertical 1' },\n { value: 'https://i.imgur.com/E0BWdxQ.jpg', label: 'Vertical 2' },\n { value: 'https://i.imgur.com/Su2OXnX.jpg', label: 'White background' },\n { value: 'https://i.imgur.com/7UnBuIf.jpg', label: 'Pile' },\n ],\n value: 'https://i.imgur.com/1T78BGV.jpg'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":826,"value":"canvas(image)","pinned":false,"hidden":false,"mode":"ojs"},{"id":516,"value":"image = {\n const img = html`

`\n return Generators.observe(notify => {\n img.addEventListener('load', event => {\n const { target } = event\n const canvas = DOM.canvas(target.width, target.height),\n ctx = canvas.getContext('2d')\n\n ctx.drawImage(target, 0, 0, canvas.width, canvas.height)\n const image = ctx.getImageData(0, 0, canvas.width, canvas.height)\n\n notify(image)\n })\n })\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":555,"value":"function canvas(image, style = 'width: 100%') {\n const canvas = DOM.canvas(image.width, image.height),\n ctx = canvas.getContext('2d')\n ctx.putImageData(image, 0, 0)\n if (style) canvas.style = style\n \n return canvas\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":615,"value":"{\n const node = slide.html('small')`\n

❓ImageData

\n `\n \n const canvas = node.querySelector('canvas'),\n ctx = canvas.getContext('2d')\n ctx.putImageData(imageWut, 0, 0)\n \n return node\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":647,"value":"imageWut = {\n const array = [\n// ↓ red: top left ↓ orange: top middle ↓ yellow: top right (wrap)\n 255, 128, 0, 255, 255, 0, 0, 255, 255, 255, 0, 255,\n 0, 255, 0, 255, 0, 0, 255, 255, 128, 0, 255, 255,\n// ↑ green: bottom left ↑ blue: bottom middle ↑ purple: bottom right\n ]\n return new ImageData(Uint8ClampedArray.from(array), 3, 2)\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":678,"value":"notes.md`\nThe image data objects we'll be using are one-dimensional arrays of red, green, blue, and alpha/transparency (RGBA) values.\n\nSo if you had just 1 pixel, you'd still have a 4-element array of RGBA values. For a 3x2 image, you get 6 pixels which means a 24-element array. You can also think of the array as naturally wrapping at the width of the image. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":70,"value":"slide.md('small')`# 🕵🏻‍♀️ Identify cards — threshold`","pinned":false,"hidden":false,"mode":"ojs"},{"id":836,"value":"{\n const result = new ImageData(image.width, image.height),\n r = result.data,\n d = image.data\n for (let i = 0; i < d.length; i += 4) {\n r[i] = r[i + 1] = r[i + 2] = luminosity(d[i], d[i + 1], d[i + 2]) >= thresholdValue ? 255 : 0\n r[i + 3] = d[i + 3]\n }\n\n return canvas(result)\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":815,"value":"viewof thresholdValue = slider({\n title: 'thresholdValue',\n value: 215,\n min: 0,\n max: 255,\n step: 1,\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":810,"value":"function luminosity(r, g, b) {\n return 0.2126 * r + 0.7125 * g + 0.0722 * b\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":813,"value":"function threshold(l) {\n const thresholdValue = 212\n return l >= thresholdValue ? 255 : 0\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":821,"value":"function thresholdImage(image) {\n const result = new ImageData(image.width, image.height),\n r = result.data,\n d = image.data\n for (let i = 0; i < d.length; i += 4) {\n r[i] = r[i + 1] = r[i + 2] = threshold(luminosity(d[i], d[i + 1], d[i + 2]))\n r[i + 3] = d[i + 3]\n }\n\n return result\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1200,"value":"thresholded = thresholdImage(image, 212)","pinned":false,"hidden":false,"mode":"ojs"},{"id":897,"value":"notes.md`\nWe go through and grayscale the entire image, using [luma][1] calculated from RGB. You'll notice that the human eye doesn't consider all red, green, and blue values uniformly bright. There are a lot of reasons for that.\n\nNext, we threshold each pixel: if the luma is greater than a certain threshold value (${thresholdValue} in this case), we set it to white and otherwise black.\n\nAll of this is in support of the next step: contour finding.\n\n[1]: https://en.wikipedia.org/wiki/Luma_(video)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":854,"value":"slide.md('small')`# 🕵🏻‍♀️ Identify cards — contour`","pinned":false,"hidden":false,"mode":"ojs"},{"id":876,"value":"{\n const c = canvas(image),\n ctx = c.getContext('2d')\n ctx.putImageData(thresholded, 0, 0)\n \n ctx.strokeStyle = 'tomato'\n ctx.lineWidth = 20\n for (const contour of cardContours) {\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n }\n \n return c\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":840,"value":"contourFinder = require('contours@1.0.1')","pinned":true,"hidden":false,"mode":"ojs"},{"id":843,"value":"cardContours = {\n const area = image.width * image.height\n return contourFinder(thresholded)\n .filter(c => c.length > 4) // large enough to have area\n .map(c => c.map(p => [p % image.width, Math.floor(p / image.width)])) // switch to [x, y]\n .map(d3.polygonHull) // convex hulls\n .filter(hull => area / 125 < d3.polygonArea(hull) && d3.polygonArea(hull) < area / 5)\n .slice(0, 12)\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":850,"value":"notes.md`\nIn looking for the cards, we need to first find possible shapes in the (thresholded) image.\n\nWe can use the [\\`contours\\`][2] library which is an implementation of [Moore neighborhood tracing][1]. This gives us a path of pixels for many of the shapes in the image.\n\nWe then simplify each contour to their convex hull using [\\`d3.polygonHull\\`][3] which implements [Andrew's monotone chain algorithm][3].\n\nFinally, we want to filter down the set of contours to ones whose areas are between 1-20% of the image. The area range is arbitrary, but chosen based on the idea that 12 cards will each take up about 8% of the image by area.\n\n[1]: http://www.imageprocessingplace.com/downloads_V3/root_downloads/tutorials/contour_tracing_Abeer_George_Ghuneim/moore.html\n[2]: https://www.npmjs.com/package/contours\n[3]: https://github.com/d3/d3-polygon#polygonHull\n[4]: https://en.wikibooks.org/wiki/Algorithm_Implementation/Geometry/Convex_hull/Monotone_chain\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"slide.md('small')`# 🃏 Extract each card`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2181,"value":"{\n const c = canvas(image),\n ctx = c.getContext('2d')\n ctx.putImageData(image, 0, 0)\n \n ctx.strokeStyle = 'tomato'\n ctx.lineWidth = 20\n for (const contour of cardRectangles) {\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n }\n \n return c\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":918,"value":"cardRectangles = cardContours.map(points => {\n let rect = points\n while (rect.length > 4) {\n const area = d3.polygonArea(rect)\n let min = Infinity,\n minRect = null\n for (let i = 0; i < rect.length; i++) {\n const smaller = rect.slice(0, i).concat(rect.slice(i + 1)),\n difference = area - d3.polygonArea(smaller)\n if (difference < min) {\n min = difference\n minRect = smaller\n }\n }\n rect = minRect\n }\n\n // try to correct vertex order (counter-clockwise from top right corner (upside down is ok))\n const edges = [\n d3.polygonLength([rect[1], rect[0]]),\n d3.polygonLength([rect[2], rect[1]]),\n ]\n if (edges[1] > edges[0])\n // need to rotate starting vertex\n rect = rect.slice(1).concat(rect.slice(0, 1))\n\n return rect\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":2188,"value":"slide.md('small')`# 🃏 Extract each card`","pinned":false,"hidden":false,"mode":"ojs"},{"id":936,"value":"html`${cardImages.map(i => canvas(i, 'width: 25%'))}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":925,"value":"perspectiveTransform = require('perspective-transform@1.1.3')","pinned":true,"hidden":false,"mode":"ojs"},{"id":923,"value":"cardImages = cardRectangles.map(rectangle => {\n const crop = 2,\n cw = 150,\n ch = (cw / 3) * 2,\n target = _.flatten([\n [cw + crop, -crop],\n [-crop, -crop],\n [-crop, ch + crop],\n [cw + crop, ch + crop],\n ])\n\n const tx = perspectiveTransform(_.flatten(rectangle), target)\n\n const card = new ImageData(cw, ch)\n const cardBuffer = new Uint32Array(card.data.buffer),\n imageBuffer = new Uint32Array(image.data.buffer)\n\n for (let i = 0; i < cardBuffer.length; i++) {\n const x = i % cw,\n y = Math.floor(i / cw),\n [xs, ys] = tx.transformInverse(x, y),\n j = Math.round(ys) * image.width + Math.round(xs)\n cardBuffer[i] = imageBuffer[j]\n }\n\n return card\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1674,"value":"notes.md`\nLet's copy these cards out of the image and correct for our non-top-down perspective.\n\nWe're going to simplify each hull down to a quadrilateral. Most straightforward approach here is incrementally removing a vertex until we get down to just four. The best next vertex to remove is the one that causes a minimal change in polygon area. There's probably an old-white-man-last-name algorithm for this, but I don't know what it is. If someone knows, please tell me.\n\nWhen we have these \"bounding boxes\", we can use [matrix math][1] to do a perspective transform (again using an aptly named npm library, [perspective-transform][2]). For each target pixel in our top-down card view, we copy in a corresponding pixel from the source image.\n\n[1]: https://en.wikipedia.org/wiki/Transformation_matrix#Perspective_projection\n[2]: https://www.npmjs.com/package/perspective-transform\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":944,"value":"slide.md('small')`# 💡 White balancing`","pinned":false,"hidden":false,"mode":"ojs"},{"id":942,"value":"html`${cardWhiteBalanced.map(i => canvas(i, 'width: 25%'))}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":939,"value":"cardWhiteBalanced = cardImages.map(image => {\n const { data } = image\n\n // average the top and bottom border for what should be white\n const whitePoint = [0, 0, 0, 255]\n for (let y of [0, image.height - 1]) {\n for (let x = 0; x < image.width; x++) {\n const i = (y * image.width + x) * 4\n for (let d = 0; d < 3; d++) whitePoint[d] += data[i + d]\n }\n }\n for (let d = 0; d < 3; d++) whitePoint[d] /= image.width * 2\n\n // adjust in RGB space\n const result = new ImageData(image.width, image.height)\n const multiplier = whitePoint.map(x => 255 / x)\n for (let i = 0; i < data.length; i += 4)\n for (let d = 0; d < 4; d++) result.data[i + d] = data[i + d] * multiplier[d]\n\n return result\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1305,"value":"notes.md`\nThe white balancing we're talking about here is correcting for warm or cool lighting in our environment. You can see in our original card images we have an orange-y tint and that's going to throw off our color and shading detection.\n\nTo fix it, we're going to assume the background is meant to be white and sample the top and bottom rows of the card, then adjust all the colors of the images (in RGB space, but there are probably more correct color spaces to do this in) so that we end up with the background closer to \\`rgb(255, 255, 255)\\`.\n\n[1]: https://en.wikipedia.org/wiki/Mae_Jemison\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2152,"value":"slide.md`\n# 🗺 The plan (recap)\n\n- Camera → video → canvas ✅\n- Identify cards ✅\n- Extract each card and fix perspective ✅\n- For each card, identify:\n - Color\n - Number\n - Shading\n - Shape\n- Brute force solve for sets\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":77,"value":"slide.md('small')`# 🖍 Color`","pinned":false,"hidden":false,"mode":"ojs"},{"id":990,"value":"vegaEmbed({\n data: {\n values: _.flatten(cardWhiteBalanced.map((image, i) => [...hues(image)].map(hue => {\n return { hue, row: Math.floor(i / 4), column: i % 4 }\n })))\n },\n mark: { type: 'tick', size: width * 0.085 },\n height: width * 0.1,\n width: width / 4.5,\n encoding: {\n x: { field: 'hue', type: 'quantitative', scale: { domain: [0, 360], nice: false }, axis: { title: null } },\n color: { field: 'hue', type: 'quantitative', scale: { domain: [0, 360], scheme: 'sinebow' }, legend: false },\n row: { field: 'row', type: 'ordinal', header: { title: null, labelFontSize: 1 } },\n column: { field: 'column', type: 'ordinal', header: { title: null, labelFontSize: 1 } },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1023,"value":"function* hues(image) {\n const { data } = image\n for (let i = 0; i < data.length; i += 4)\n if (threshold(luminosity(data[i], data[i + 1], data[i + 2])) === 0)\n yield d3.hsl(d3.rgb(data[i], data[i + 1], data[i + 2])).h\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":984,"value":"hue = d3.scaleThreshold()\n .domain([70, 180, 340])\n .range(['red', 'green', 'purple', 'red'])","pinned":true,"hidden":false,"mode":"ojs"},{"id":986,"value":"cardColors = cardWhiteBalanced.map(image => {\n const counts = { red: 0, green: 0, purple: 0 }\n\n for (const h of hues(image))\n counts[hue(h)]++\n const space = counts.red + counts.green + counts.purple\n\n // return counts\n if (counts.purple / space > 0.2) return 'purple'\n else if (counts.green / space > 0.5) return 'green'\n else if (counts.red / space > 0.1) return 'red'\n else return null\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1307,"value":"notes.md`\nFor each pixel (ignoring white), look at the hue distribution.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":78,"value":"slide.md('small')`# 2️⃣ Number`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1996,"value":"html`

\n${[6, 1, 0].map(i => {\n const c = canvas(cardWhiteBalanced[i], 'width: 25%; border: solid 2px hsl(0, 0%, 53%); margin: 5px;'),\n ctx = c.getContext('2d')\n\n ctx.strokeStyle = 'black'\n ctx.lineWidth = 3\n for (const contour of cardShapeContours[i]) {\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n }\n \n return c\n})}\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1370,"value":"vegaEmbed({\n data: { values: cardPercentageWidths.map((width, i) => ({ i, width, number: numberScale(width) })) },\n mark: { type: 'tick', thickness: 3 },\n width: width * 0.95,\n autosize: 'fit-x',\n encoding: {\n x: { field: 'width', type: 'quantitative', scale: { domain: [0, 1] }, axis: { format: '%' } },\n y: { field: 'number', type: 'ordinal', scale: { domain: _.filter(numberScale.range()), rangeStep: width * 0.08 } },\n detail: { field: 'i', type: 'ordinal' },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1191,"value":"cardShapeContours = cardImages.map(image => {\n const area = image.width * image.height\n return contourFinder(thresholdImage(image))\n .filter(c => c.length > 4) // large enough to have area\n .map(c => c.map(p => [p % image.width, Math.floor(p / image.width)])) // switch to [x, y]\n .filter(hull => area / 10 < Math.abs(d3.polygonArea(hull)) && Math.abs(d3.polygonArea(hull)) < area / 5)\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1362,"value":"numberScale = d3.scaleThreshold()\n .domain([0.1, 0.44, 0.75, 0.99])\n .range([null, 1, 2, 3, null])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1364,"value":"cardPercentageWidths = _.zip(cardShapeContours, cardImages).map(([contours, {width}]) => {\n let min = width,\n max = 0\n\n for (const shape of contours) {\n for (const [x, y] of shape) {\n if (x < min) min = x\n if (x > max) max = x\n }\n }\n\n // mirror a min or max to the other side for incomplete shape finding\n if (width - min > max) max = width - min\n if (width - max < min) min = width - max\n\n return (max - min) / width\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1366,"value":"cardNumbers = cardPercentageWidths.map(numberScale)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1309,"value":"notes.md`\nWe're going to need the shape contours coming up. Same idea as before, but applied to each card image.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":79,"value":"slide.md('small')`# 🌴 Shading`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1882,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":1270,"value":"vegaEmbed({\n data: { values: cardInteriorLumas.map((luma, i) => ({ luma, i, shade: shadeScale(luma) })) },\n mark: { type: 'tick', thickness: 3 },\n width: width * 0.95,\n autosize: 'fit-x',\n encoding: {\n x: { field: 'luma', type: 'quantitative', scale: { domain: [0, 255], nice: false } },\n y: { field: 'shade', type: 'nominal', scale: { domain: shadeScale.range(), rangeStep: width * 0.08 } },\n detail: { field: 'i', type: 'nominal' },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1248,"value":"shadeScale = d3.scaleThreshold()\n .domain([180, 240])\n .range(['solid', 'striped', 'outlined'])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1241,"value":"cardInteriorLumas = _.zip(cardWhiteBalanced, cardShapeContours).map(([image, contours]) => {\n const { data } = image\n let sum = 0, n = 0\n\n for (const shape of contours) {\n // sample interior\n const xExtent = d3.extent(shape, p => p[0]),\n yExtent = d3.extent(shape, p => p[1]),\n dx = Math.floor((xExtent[1] - xExtent[0]) / 3),\n dy = Math.floor((yExtent[1] - yExtent[0]) / 4)\n\n for (let y = yExtent[0] + dy; y < yExtent[1] - dy; y++) {\n for (let x = xExtent[0] + dx; x < xExtent[1] - dx; x++) {\n const i = (y * image.width + x) * 4\n sum += luminosity(data[i], data[i + 1], data[i + 2])\n n++\n }\n }\n }\n\n return sum / n\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1259,"value":"cardShades = cardInteriorLumas.map(shadeScale)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1311,"value":"notes.md`\nSample the interior of each shape inside a card. Average the luma inside of the shape. Solid keeps to a pretty dark luma, outline is near-white, and striped is in the middle.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":80,"value":"slide.md('small')`# 💎 Shape`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2232,"value":"notes.md`## Area of contour / Area of bounding box`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2064,"value":"html`

\n${[10, 8, 9].map(i => {\n const c = canvas(cardWhiteBalanced[i], 'width: 25%'),\n ctx = c.getContext('2d')\n\n ctx.strokeStyle = 'black'\n ctx.lineWidth = 2\n for (const contour of cardShapeContours[i]) {\n ctx.strokeStyle = 'black'\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n \n const xExtent = d3.extent(contour, p => p[0]),\n yExtent = d3.extent(contour, p => p[1])\n ctx.strokeStyle = 'tomato'\n ctx.beginPath()\n ctx.lineTo(xExtent[0], yExtent[0])\n ctx.lineTo(xExtent[1], yExtent[0])\n ctx.lineTo(xExtent[1], yExtent[1])\n ctx.lineTo(xExtent[0], yExtent[1])\n ctx.closePath()\n ctx.stroke()\n }\n \n return c\n})}\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1390,"value":"vegaEmbed({\n data: { values: _.zip(cardShapeAreas, cardHullAreas, cardShapes).map(([area, hull, shape], i) => ({ area, hull, shape, i })) },\n mark: { type: 'point', thickness: 3 },\n width: width * 0.95,\n height: width * 0.1,\n autosize: 'fit-x',\n encoding: {\n x: { field: 'area', type: 'quantitative', axis: { title: 'area of bounding box', format: '%' } },\n // y: { field: 'hull', type: 'quantitative', axis: { title: 'area of convex hull', format: '%' } },\n shape: { field: 'shape', type: 'nominal', scale: { domain: ['diamond', 'oval', 'squiggle'], range: ['diamond', 'circle', 'square'] } },\n color: { field: 'shape', type: 'nominal', scale: { domain: ['diamond', 'oval', 'squiggle'] } },\n size: { value: width * 0.3 },\n detail: { field: 'i', type: 'nominal' },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2230,"value":"notes.md`## Area of contour / Area of convex hull`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1839,"value":"html`

\n${[10, 8, 9].map(i => {\n const c = canvas(cardWhiteBalanced[i], 'width: 25%'),\n ctx = c.getContext('2d')\n\n ctx.strokeStyle = 'black'\n ctx.lineWidth = 2\n for (const contour of cardShapeContours[i]) {\n ctx.strokeStyle = 'black'\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n \n const xExtent = d3.extent(contour, p => p[0]),\n yExtent = d3.extent(contour, p => p[1]),\n dx = Math.floor((xExtent[1] - xExtent[0]) / 3),\n dy = Math.floor((yExtent[1] - yExtent[0]) / 4)\n ctx.strokeStyle = 'tomato'\n ctx.beginPath()\n for (const [x, y] of d3.polygonHull(contour)) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n }\n \n return c\n})}\n

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2246,"value":"vegaEmbed({\n data: { values: _.zip(cardShapeAreas, cardHullAreas, cardShapes).map(([area, hull, shape], i) => ({ area, hull, shape, i })) },\n mark: { type: 'point', thickness: 3 },\n width: width * 0.95,\n height: width * 0.3,\n autosize: 'fit-x',\n encoding: {\n x: { field: 'area', type: 'quantitative', axis: { title: 'area of bounding box', format: '%' } },\n y: { field: 'hull', type: 'quantitative', axis: { title: 'area of convex hull', format: '%' } },\n shape: { field: 'shape', type: 'nominal', scale: { domain: ['diamond', 'oval', 'squiggle'], range: ['diamond', 'circle', 'square'] } },\n color: { field: 'shape', type: 'nominal', scale: { domain: ['diamond', 'oval', 'squiggle'] } },\n size: { value: width * 0.3 },\n detail: { field: 'i', type: 'nominal' },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1384,"value":"shapeCentroids = [\n { shape: 'diamond', center: [0.53, 0.062] },\n { shape: 'squiggle', center: [0.715, 0.145] },\n { shape: 'oval', center: [0.85, 0.026] },\n]","pinned":true,"hidden":false,"mode":"ojs"},{"id":1620,"value":"function polygonBox(points) {\n return _.zip(d3.extent(points, ([x, y]) => x), d3.extent(points, ([x, y]) => y))\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1630,"value":"function polygonBoxArea(points) {\n const [min, max] = polygonBox(points)\n return (max[0] - min[0]) * (max[1] - min[1])\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1386,"value":"cardShapeAreas = _.zip(cardShapeContours, cardImages).map(([contours, image]) => {\n return d3.mean(contours.map(contour => Math.abs(d3.polygonArea(contour)) / polygonBoxArea(contour)))\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1421,"value":"cardHullAreas = cardShapeContours.map(contours => {\n return _.mean(contours.map(contour => {\n const hullArea = Math.abs(d3.polygonArea(d3.polygonHull(contour)))\n return (hullArea - Math.abs(d3.polygonArea(contour))) / hullArea\n }))\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1396,"value":"cardShapes = _.zip(cardShapeAreas, cardHullAreas).map(point => {\n const squares = _.sortBy(shapeCentroids, ({ center }) => {\n const d = [point[0] - center[0], point[1] - center[1]]\n return d[0] * d[0] + d[1] * d[1]\n })\n return squares[0].shape\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":76,"value":"slide.md`\n# 💪🏼 Brute force set finding\n\n- 12 cards with 3 cards per set = 220 possible sets\n- this part literally takes ${(ms * 100).toFixed(1)} µs\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2107,"value":"slide.md('small')`# 🏎 Sets per minute`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1836,"value":"vegaEmbed({\n data: { values: [\n { name: 'Visnu', perMinute: 1 },\n { name: 'Susan', perMinute: 20 },\n howFastIsCompy ? { name: 'Computer', perMinute: 60000 / ms } : {},\n ] },\n mark: { type: 'bar', thickness: 2 },\n width: width * 0.95,\n height: width * 0.3,\n autosize: 'fit-x',\n encoding: {\n y: { field: 'name', type: 'nominal', scale: { domain: [ 'Visnu', 'Susan', 'Computer' ] }, axis: { title: null } },\n x: { field: 'perMinute', type: 'quantitative', axis: { title: 'Sets per minute' } },\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2096,"value":"viewof howFastIsCompy = checkbox({\n options: [{ value: \"toggle\", label: \"Show me the carnage!\" }],\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"combinations = require('https://bundle.run/combinations-generator@1.0.1')","pinned":true,"hidden":false,"mode":"ojs"},{"id":1663,"value":"possibleSets = [...combinations(_.range(0, 12), 3)]","pinned":false,"hidden":false,"mode":"ojs"},{"id":1492,"value":"function isSet(cards) {\n return [ cardColors, cardNumbers, cardShades, cardShapes ].every(attribute => {\n attribute = cards.map(i => attribute[i])\n return allSame(attribute) || allDifferent(attribute)\n })\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1488,"value":"function allSame(attributes) {\n return attributes[0] === attributes[1] &&\n attributes[1] === attributes[2]\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1490,"value":"function allDifferent(attributes) {\n return attributes[0] !== attributes[1] &&\n attributes[0] !== attributes[2] &&\n attributes[1] !== attributes[2]\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":293,"value":"sets = possibleSets.filter(isSet)","pinned":true,"hidden":false,"mode":"ojs"},{"id":287,"value":"ms = {\n const t0 = performance.now()\n ;possibleSets.filter(isSet)\n return performance.now() - t0\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"notes.md`\n12 cards in sets of 3? That's all? The math is ${tex`{12 \\choose 3} = \\frac{12!}{3!(12-3)!} = \\frac{12 \\cdot 11 \\cdot 10}{3 \\cdot 2} = 220`}. A computer can do that in no time. And when I say \"no time\" I mean ${(ms*100).toFixed(1)} µs is exactly how long that code took to run just now. That's the entire point of this exercise.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"slide.md('small')`# 🎨 Redraw on source image`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1498,"value":"result = {\n const c = canvas(image),\n ctx = c.getContext('2d')\n \n ctx.lineWidth = 20\n let n = 0\n for (const set of sets) {\n ctx.strokeStyle = d3.schemeCategory10[n++ % 10]\n for (const contour of set.map(i => cardContours[i])) {\n ctx.beginPath()\n for (const [x, y] of contour) ctx.lineTo(x, y)\n ctx.closePath()\n ctx.stroke()\n }\n }\n \n return c\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":88,"value":"slide.html('bleed')`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1750,"value":"slide.md`\n# 😥 We lost!\n\n- False positives\n- Lighting & shadows\n- Overfitting!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2302,"value":"notes.md`\nOMG! We still lost! Why? Well, we built this whole thing based on 1-2 images. That is a classic overfitting faux paux.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":93,"value":"slide.md`\n## 💯 Hella source images\n\n- Mean absolute error\n- Picking magic numbers using more data\n - \"Machine learning\"\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2304,"value":"notes.md`\nSo, what do we do? Get a lot of source images! I went back and built in a debug mode to my runtime version which sends frames to a server and records which cards it thinks is in the frame. Then using a debugging UI I can quickly tag which frames are correct and quickly build up a training set. We can use all of these source images to tweak all of those magic numbers and thresholds using a lot more experience than just 1-2 images. I call this \"poor man's machine learning\" which really is... just what machine learning is.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":90,"value":"slide.html('bleed')`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2308,"value":"notes.md`\nTo fast forward through the training process, I've put together a classic Rocky-style \"training\" montage.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1708,"value":"slide.md`\n# ✨ Demo\n- Guest starring Anne\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":98,"value":"slide.md`\n# 🎉 We won!\n\nWe actually built something that works!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1704,"value":"slide.md`\n# 😥 We lost!\n\nWe actually built something that _almost_ works!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":96,"value":"slide.html('bleed')`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1601,"value":"slide.md`\n# 📥 TODO\n\n- WebGL\n - Shaders + GLSL\n- Tensorflow.js\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1606,"value":"notes.md`\nThere's still plenty to do.\n\nDoing all of this computation in CPU rather than GPU (which is really, really good at this type of stuff) would be great. It's also an excuse to learn [GLSL][1].\n\nAnd now that I've started building up a library of tagged images, I can unlock f'real machine learning and let something like [Tensorflow.js] figure this out for me.\n\n[1]: https://thebookofshaders.com/\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":100,"value":"slide.md`\n# Other things to cheat with\n\n- [Imperial Assault][1]\n- [Medium's Snowflake][2]\n\n[1]: https://beta.observablehq.com/@visnup/imperial-assault\n[2]: https://snowflake.medium.com\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1804,"value":"notes.md`\nA few other examples of what I call cheating.\n\nIn Imperial Assault a somewhat permanent choice you need to make when building your character is what items to buy and bring. But how do you really compare them? You can play a lot or you can simulate!\n\nI should also point out examples like AlphaZero which some see as _expanding_ our understanding of go and chess via novel new play humans haven't yet considered.\n\nFinally, I want to draw attention to a tool I _love_ from Medium called snowflake. It's a tool to see where you stand on their (or your) engineering ladder. It's like a worksheet to play with how you can get better as an engineer. It helps target your 1:1s with your manager or mentor and helps you highly focus and target your efforts on improving your skills. For all the times you've wondered to yourself, \"what can I work on to get better as an engineer?\" or \"what would concentrating on this specific area look like?\" this is the answer staring you in the face.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1810,"value":"slide.md`\n# Why?\n\n- Cheating, hacking is an essential life skill\n- Mischief → Innovation\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":102,"value":"notes.md`\nWhy are we talking about all this again?\n\nI personally think of cheating, hacking, or solving problems creatively as an essential life skill. Some people call it innovation. It’s why humans have succeeded so far (for better or worse).\n\n[image: technology curve]\n\n[image: geological timescale]\n\nIf you look at the human population explosion (as a indication of increasing technology aptitude) and where we sit on the geological timescale of the Earth, I just keep thinking _technology is truly a wonder_. On a [geological timescale](https://en.wikipedia.org/wiki/Geologic_time_scale), it's hard for me not to think that human beings are ultimately screwed when going up against entropy and time, but cheating, I truly believe, is going to be the thing that gets us past that.\n\nTo me, innovation, technology, building tools are all borne out of a sense of mischieviousness.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":110,"value":"slide.md`# ❓ What will you build?`","pinned":false,"hidden":false,"mode":"ojs"},{"id":329,"value":"slide.md`\n## 👉🏼 [observablehq.com/@visnup/cheat](https://beta.observablehq.com/@visnup/cheat)\n\nThis notebook can be read blog-like or presentation-like. Share with your friends!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1526,"value":"md`---\n## Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":296,"value":"_ = require('lodash@4')","pinned":false,"hidden":false,"mode":"ojs"},{"id":845,"value":"d3 = require('d3-array@2', 'd3-color@1', 'd3-polygon@1', 'd3-scale@2', 'd3-scale-chromatic@1')","pinned":false,"hidden":false,"mode":"ojs"},{"id":998,"value":"vegaEmbed = require('vega-embed@3')","pinned":false,"hidden":false,"mode":"ojs"},{"id":520,"value":"import { checkbox, select, slider } from '@jashkenas/inputs'","pinned":false,"hidden":false,"mode":"ojs"},{"id":2345,"value":"import { slide, notes, keyBindings, style } from '@visnup/slides'","pinned":false,"hidden":false,"mode":"ojs"},{"id":2354,"value":"style","pinned":false,"hidden":false,"mode":"ojs"},{"id":2355,"value":"keyBindings","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}