31:T5ea, ### Fun with image analysis Our first attempt was to try to identify the duplicated avatars directly. Because we re-encode images during our upload process, we couldn’t do an exact byte-for-byte match. Instead we looked towards perceptual image hashing. Using the [image-hash](https://www.npmjs.com/package/image-hash) library on NPM we calculated a short code for every image. These codes had the property that small changes to the code would correspond with small changes to the image. With these codes in hand we started tuning. Loading the avatars into a notebook, we looked for a set of thresholds and settings that produced the cut of the data we wanted to identify only the duplicate images. Unfortunately, every combination we worked with had an unacceptably high error rate. Avatars that weren’t duplicated were getting flagged, and avatars that were clearly duplicates were going unmatched. Our next attempt was to identify some of the images we knew weren’t duplicated. We knew that any automatically generated avatar would be unique, so we built a classifier to look for those using an edge detection algorithm, [the Sobel operator](https://observablehq.com/@mbostock/sobel-operator). This produced a fascinating notebook, and showed us that the majority of the potentially affected users had an avatar supplied by their authentication provider. That meant that the sample of users we were looking for, those where we didn’t need to reset their avatar, was too small a group to worry about.32:[{"id":"58291532cc5ad34e3c1bc25274b23ab37f6499d0c29187572c64364d901c2b499d753d0c76d52894933d130b56caad04efef81763ef49b54f8f58db1194bdbcf","href":"https://static.observableusercontent.com/files/58291532cc5ad34e3c1bc25274b23ab37f6499d0c29187572c64364d901c2b499d753d0c76d52894933d130b56caad04efef81763ef49b54f8f58db1194bdbcf","name":"demo-avatar-1.jpg","type":"image/jpeg","lastModified":1655158209983.973,"size":8051},{"id":"2c9bc7942be2123132c0ec1bc7ff8b78c38c4ae6e2cf3fe827bd6ea8807f9813a08a04c44b7fbcba9c0d1f727cefad5fd09827e065aaaa1257aa8d73eea936b8","href":"https://static.observableusercontent.com/files/2c9bc7942be2123132c0ec1bc7ff8b78c38c4ae6e2cf3fe827bd6ea8807f9813a08a04c44b7fbcba9c0d1f727cefad5fd09827e065aaaa1257aa8d73eea936b8","name":"demo-avatar-4.jpg","type":"image/jpeg","lastModified":1655158209983.973,"size":1416},{"id":"39f603b9735028b224ebefea52ad9de7ac367272bb036e6e7fc6d0e00fbed97e71ba1a4dbd2d1782e76d10b82d1ada7360d3b9a1f3d5db0c27dab31527c71ea4","href":"https://static.observableusercontent.com/files/39f603b9735028b224ebefea52ad9de7ac367272bb036e6e7fc6d0e00fbed97e71ba1a4dbd2d1782e76d10b82d1ada7360d3b9a1f3d5db0c27dab31527c71ea4","name":"wayne-avatar.jpg","type":"image/jpeg","lastModified":1655158209983.973,"size":18425}] 2f:{"title":"Duplicate avatars retrospective","cells":[{"id":0,"value":"# Duplicate avatars retrospective\n## affecting sign ups from May 18 to June 3","pinned":false,"hidden":false,"mode":"md"},{"id":36,"value":"We recently improved [our sign-up process](https://observablehq.com/@observablehq/signing-up), giving users a chance to set up their public profile while creating their account. Users now have the option to set a display name and avatar during their sign up process, instead of starting their journey as an anonymous string of hex digits and a grey circle avatar. The avatar can be supplied by an authentication provider, such as Google, or generated by Observable.","pinned":false,"hidden":false,"mode":"md"},{"id":447,"value":"Unfortunately, there was a bug in the way we handled avatars. Sometimes when we loaded a third party avatar from Google, Microsoft, or Twitter, we’d use a cached version of another user that had recently signed up. This resulted in a number of users having duplicated avatars. To resolve this issue, we eventually ended up resetting the avatars of potentially affected users.","pinned":false,"hidden":false,"mode":"md"},{"id":238,"value":"sampleAvatars = {\n const count = 5;\n const margin = 8;\n const size = (Math.min(width, 640) - margin * (count - 1)) / count;\n const style={marginRight: `${margin}px`, borderRadius: `${size / 2}px`, overflow: 'hidden'};\n\n const avatars = d3.range(count).map((d) => htl.html`

${genAvatar({size})}

`)\n \n while (true) {\n let idx = Math.floor(Math.random() * count);\n avatars[idx] = htl.html`

${genAvatar({size})}

`;\n yield htl.html`

${avatars}

Some examples of the random avatars we generate for users.

\n`;\n await Promises.delay(4000)\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":43,"value":"## Timeline (PDT)","pinned":false,"hidden":false,"mode":"md"},{"id":224,"value":"### May 18, 16:30 - Bug introduced\nWe enabled the new flow for all sign ups, and began serving sign up avatars with cache headers. Our content delivery network (CDN) respected these headers, and started caching and re-serving those avatars.\n","pinned":false,"hidden":false,"mode":"md"},{"id":226,"value":"### June 2, 9:30 - Bug reported\nWe received a report that a user had signed up and had an automatic avatar provided that was a photo of another person. Knowing that these kinds of issues rarely affect just one user, we built an Observable notebook that showed the avatars of recent sign ups. It didn’t take long to find another user that had the same avatar, and to notice that many other users had duplicate avatars as well.\n\nAt this point we learned that the problem was caused by incorrectly forwarding the cache headers of third party avatars. During the sign up process, third party avatars are proxied through a single endpoint on our server. Our CDN dutifully used these cache headers to serve new users with the cached avatars.\n","pinned":false,"hidden":false,"mode":"md"},{"id":228,"value":"### June 2, 13:00 - Stopped sending cache headers\nWe deployed a new version that removed the cache headers for the avatar request. The response then used our default cache behavior for the API of never caching data. In theory this should have fixed the problem, and had our original deployment used this code, we would have never had this problem. But we still had more work to do to clean up our problem.\n","pinned":false,"hidden":false,"mode":"md"},{"id":230,"value":"### June 2, 18:00 - Cleared our CDN cache\nSince our “new user avatars” notebook was still showing duplicates, we purged the cache for the affected endpoint. We expected this to clear up the problem, and left it running for the night to gather data.\n","pinned":false,"hidden":false,"mode":"md"},{"id":61,"value":"### June 3, 15:00 - Updated URLs\nClearing the cache helped the problem, but ultimately the long tail of caches on the internet goes on forever. Our only remaining choice was to make sure every avatar URL was unique and uncacheable. We added a SHA2 hash of the inputs as a query string to the avatar URL. To make sure that we didn’t have any more lurking problems, we also configured the system to check that the provided hash matched the avatar we returned to users.\n","pinned":false,"hidden":false,"mode":"md"},{"id":232,"value":"### June 6, 14:00 - Repair and communication\nWe knew that no more new users were getting duplicate avatars, but we still had many users that had duplicate avatars. Ultimately we decided that the only solution we’d be happy with was to generate new random avatars for any user that may have been exposed to the duplicate avatar solution. We reset the affected avatars, and notified the users that they may want to select a new avatar.","pinned":false,"hidden":false,"mode":"md"},{"id":301,"value":"## Exploring the problem\n\nWe knew from the start that the solution would be to reset the avatars of some users, but wanted to make this set of users as small as possible. We started with a large group of potentially impacted users and as we learned more about the timeline of the bug and its exact behavior, we were eventually able to narrow that down to 18% of the original group of users.\n","pinned":false,"hidden":false,"mode":"md"},{"id":153,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":418,"value":"Here are some examples of avatars you’ll find around the site, and the result of applying the Sobel operator to them.","pinned":false,"hidden":false,"mode":"md"},{"id":322,"value":"

\n ${(await Promise.all(demoAvatarUrls.map(async ({url, title}) => \n htl.html`\n

${title}

\n ${convolveY2(await createContext(htl.html`

`), Kx, Ky)}\n

`\n ))).flat()}\n

","pinned":false,"hidden":false,"mode":"html"},{"id":303,"value":"\n### Resolution\nFrom the beginning, we kept in mind the impact of this problem. Logging into a site and having the wrong avatar is a mildly strange experience, but browsing around Observable and seeing another user that has your personal avatar would be terrible. We wanted to eliminate that possibility.\n\nOur attempts to automatically find duplicate avatars didn’t produce a solid enough guarantee that we could do that, and our image analysis notebooks showed that the problem was very widespread. In the end we decided to reset the avatar of any user that could have experienced the bug, whether or not their avatar had been duplicated. This wasn’t our first choice, but it was the compromise we arrived at because we valued having a very low false negative rate more than having a low false positive rate.","pinned":false,"hidden":false,"mode":"md"},{"id":90,"value":"## Gratitude\n\nThanks to the user who notified us of this problem. And thanks to all of our new users impacted by this problem for your patience.","pinned":false,"hidden":false,"mode":"md"},{"id":97,"value":"blogSubscribeFooter","pinned":false,"hidden":false,"mode":"ojs"},{"id":47,"value":"

\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":95,"value":"import {blogSubscribeFooter} from \"615860c65e5663a6\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":315,"value":"import {convolveY2, Kx, Ky} with {avatarSize as width, avatarSize as height} from \"@mbostock/sobel-operator\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":352,"value":"avatarSize = Math.floor((Math.min(640, width) - (16 * 3)) / 8) * 2","pinned":true,"hidden":false,"mode":"ojs"},{"id":203,"value":"function genAvatar({size = 64} = {}) {\n const c = document.createElement(\"canvas\");\n c.width = size;\n c.height = size;\n const ctx = c.getContext(\"2d\");\n\n const h1 = Math.floor(Math.random() * 360);\n const hDelta = Math.random() * 40 + 20;\n const h2 = h1 > 300 ? h1 - hDelta : h1 + hDelta;\n const s = Math.floor(Math.random() * 40) + 40;\n const l1 = Math.floor(Math.random() * 30) + 30;\n const l2 = Math.floor(Math.random() * 30) + 50;\n\n const r = size / 2;\n const ang = Math.random() * Math.PI * 2;\n const cx = (Math.cos(ang) + 1) * r;\n const cy = (Math.sin(ang) + 1) * r;\n const gradCenter = [cx, cy, size * 0.1];\n const gradEdge = [cx, cy, size * 1.1];\n\n const gradient = ctx.createRadialGradient(...gradCenter, ...gradEdge);\n gradient.addColorStop(0, `hsl(${h1}, ${s}%, ${l1}%`);\n gradient.addColorStop(1, `hsl(${h2}, ${s}%, ${l2}%`);\n\n ctx.fillStyle = gradient;\n ctx.fillRect(0, 0, size, size);\n\n return c;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":330,"value":"demoAvatarUrls = [\n {title: \"Cartoon avatar\", url: await FileAttachment(\"demo-avatar-1.jpg\").url()},\n {title: htl.html`@waynesutton`, url: await FileAttachment(\"wayne-avatar.jpg\").url()},\n {title: \"Generated avatar\", url: await genAvatar({size: avatarSize}).toDataURL(\"image/png\")},\n {title: \"Google avatar\", url: await FileAttachment(\"demo-avatar-4.jpg\").url()},\n]","pinned":true,"hidden":false,"mode":"ojs"},{"id":468,"value":"async function createContext(image) {\n await new Promise((resolve) => {\n if (image.naturalWidth) resolve();\n function onLoad() { resolve() }\n image.addEventListener('load', onLoad);\n invalidation.then(image => removeEventListener('load', onLoad));\n });\n const context = DOM.context2d(avatarSize, avatarSize, 1);\n context.canvas.style = \"max-width: 100%;\";\n context.drawImage(image, 0, 0, avatarSize, avatarSize);\n return context;\n}","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}