31:T4f6,md`

[Mean, Median, and Moose](https://meanmedianmoose.ca/) and [Hackforge](http://hackf.org/) present a tutorial on [DeckGL](https://deck.gl/). DeckGL allows people to build fast and powerful maps quickly. It was originally developed by Uber and then open sourced. In this tutorial we'll take you through how to set up DeckGL in an Observable notebook and then build some visualizations. This information was originally presented at a Hackforge event. You can find a recording [here](https://www.youtube.com/watch?v=sOzu2PLKGs4). The data set we'll be using is courtest of the City of Windsor, Ontario. They post [datasets about their snow plow activity](https://opendata.citywindsor.ca/opendata/details/224) on their [open data portal](https://opendata.citywindsor.ca/). Observable notebooks already have the code and data included. What this tutorial attempts to do is also provide a textual description of the code and explanations for why the code was written the way it was. If you enjoy this kind of thing you may be interested in the [Mean, Median, and Moose podcast and accompanying blog](https://meanmedianmoose.ca/). `32:Tac8,md`The next thing we're going to do is create a map. We'll wrap this code in a function called getDeckGLMap because we'll be using the same code over and over again to generate the same map with different layers each time. The function looks like this: \`\`\`js getDeckGLMap = (layers, pitch) => { if (pitch === undefined) { pitch = 0; } let container = html \`

\`; let deckgl = new deck.DeckGL({ container, map: mapboxgl, mapboxAccessToken: '', mapboxApiAccessToken: token, mapStyle: 'mapbox://styles/mapbox/dark-v10', layers: layers, initialViewState: { longitude: -82.9835, latitude: 42.2767, zoom: 11.5, pitch: pitch }, controller: true }); return container; } \`\`\` You can see this same code and edit it by expanding the "getDeckGLMap" code block below. Let's walk through this and see what's happening. The first thing we do is create the function which takes a layers and pitch parameter. We always need a set of layers, but we make the pitch parameter optional by providing a default of zero. \`\`\`js getDeckGLMap = (layers, pitch) => { if (pitch === undefined) { pitch = 0; } [...] } \`\`\` The next thing we do is create a container for the map to be placed in. If you were using DeckGL elsewhere this would look a little different. It's commonly used in ReactJS where DeckGL just becomes a component. In an Observable notebook however, you can define an HTML block as a variable. We then return this element at the end of the function. \`\`\`js let container = html \`

\`; \`\`\` Now we define what map gets displayed in the container by defining a DeckGL component. \`\`\`js let deckgl = new deck.DeckGL({ container, map: mapboxgl, mapboxAccessToken: '', mapboxApiAccessToken: token, mapStyle: 'mapbox://styles/mapbox/dark-v10', layers: layers, initialViewState: { longitude: -82.9835, latitude: 42.2767, zoom: 11.5, pitch: pitch }, controller: true }); \`\`\` The container variable above is the html element we create in the previous step with the magic Observable definiton. We pass the mapbox library into the map parameter and provide a Mapbox access token (you should go get your own if you're going to do this by the way - my tokens are limitted and change often). mapStyle defines the mapbox theme to use. Take a look at all the styles under options.style in [the mapbox documentation](https://docs.mapbox.com/mapbox-gl-js/api/map/) to see your other options. Then we define the layers - we'll talk about those coming up. Finally, we define the initial view state. That is the initial lat/lon of the center of the mpa, a zoom level, and the pitch. `33:T726,md`Now that we have the data, let's map it! We are going to use the function we created earlie to create a DeckGL map and pass it a layer. The layer defines the data to display on the map and how to display it. We're going to start out basic and just plot every point on the map as a small orange circle. If you want to plot circles, the [DeckGL ScatterplotLayer](https://deck.gl/docs/api-reference/layers/scatterplot-layer) is the right choice. \`\`\`js getDeckGLMap([ new deck.ScatterplotLayer({ id: 'scatterplot-layer', data: data, pickable: true, opacity: 1, filled: true, radiusScale: 1, radiusMinPixels: 1, radiusMaxPixels: 1, getPosition: d => [d.Longitude, d.Latitude], getRadius: d => 1, getFillColor: d => [255, 140, 0] }) ]) \`\`\` As you can see, a layer is no more than a bag of parameters and the occasional lambda (anonymous function definition). There are two sets of parameters. The properties are things like opacity, whether the circles are filled, and the radius scale. The other kind of parameter is a data accessor. They're given away above by being prefixed with "get" (getPosition, getRadius, getFillColor). Properties define attributes of the entire layer. Data accessors define what each individual item in the "data" property looks like. They're defined as functions that are called once for every element in the array defined in the data property. getRadius and getFillColor are pretty boring - all of our circles are orange with a radius of 1 meter. getPosition puts them in the right place though by returning an array acting as a long/lat coordinate. Want to know about other properties or get a more technical definition of each one? Take a look at the [ScatterplotLayer documentation](https://deck.gl/docs/api-reference/layers/scatterplot-layer) `34:T609,md`A map with data! That's great. One problem with scatterplots though is that people can have trouble discerning details about how frequently a snow plow might be visiting a street or intersection. We can do better by making this into a heatmap instead. How do we do that? Well, with the [Heatmap layer](https://deck.gl/docs/api-reference/aggregation-layers/heatmap-layer)! \`\`\`js getDeckGLMap([ new deck.HeatmapLayer({ id: 'heatmapLayer', data, getPosition: d => [d.Longitude, d.Latitude], aggregation: 'SUM', radiusPixels: 10, colorDomain: [100, 1000] }) ]) \`\`\` Simple enough. The map is split into slightly overlaping circles with a radius of ten pixels. The more points in a circle the redder it looks. colorDomain defines the minimum and maximum values for colours. Here, circles with fewer than 100 points fade out. Circles with 1,000 points or more get the highest colour values - effectively treating all circles with 1,000 or more points the same on the map. I do that because while the defaults can be pretty good for the heatmap layer, but I've adjusted them in this case mainly to prevent plow starting points (city plow storage and service lots) from being the only red circles on the map. The heatmaps look good and give you a feeling of which streets are plowed the most in a period. This is probably why the in-built visualization on the City of Windsor open data portal uses a heatmap. Zoom in and out of the map by scrolling while your mouse pointer is over it and see how the map changes. `35:T78e,md `Okay! So we have some idea of where and how much the snowplows plow in aggregate. But what if you're interested in individual plow paths? Can we visualize those? Indeed we can! First we'll need to build a path (an array of successive coordinates) for each truck. I did this with a call to reduce on the data array, but you can iterate through it as well instead. \`\`\`js let reduced = data.reduce( (accum, d) => { if(accum[d.Truck] === undefined){ accum[d.Truck] = { Truck: d.Truck, path: [], timestamps: [] }; } accum[d.Truck].path.push([d.Longitude, d.Latitude]) accum[d.Truck].timestamps.push(new Date(d["Date time"]).getTime()) return accum; }, {}); \`\`\` All this does is build a map (an object) of all the trucks and then add a path and timestamp array for each truck. The path array contains all the coordinates for the trucks over time. The timestamps array contains the corresponding timestamps for the trucks. We also want to define a different color for each truck. The are about 230 so that can be tough. Luckily d3 can help with its Rainbow color scale - we just have to use it to also convert from the default hex colours to an array based colour that DeckGL consumes. \`\`\`js getColorScale = (min, max) => { let func = d3 .scaleSequential() .domain([min, max]) // Domain is 0 to 231 (the number of trucks) .interpolator(d3.interpolateRainbow); // Interpolate on the Rainbow color scale return d => { // Quick function to convery the hex color to an RGB array let col = d3.color(func(d)); return [col.r, col.g, col.b]; } } let colorScale = getColorScale(0,Object.keys(reduced).length); let trucks = []; let num = 0; for(let truckid in reduced){ reduced[truckid].color = colorScale(num); // Assign the truck a colour based on its index trucks.push(reduced[truckid]) num++; } \`\`\` `36:T42c,md `That's a lot of trucks! We can't really see what's going on. Luckily DeckGL has something called a [Trips layer](https://deck.gl/examples/trips-layer/) to better show trips taken by a fleet of vehicles. Makes sense considering DeckGL was originally developed by Uber! To make this happen right we'll need a slider (or scrubber) to use to travel through time. We'll also need to change the map to take updates from that scrubber without reevaluting the entire cell. First things first, let's create a container for this special map in its own Observable cell: \`\`\`js tripcontainer = html\`

\`; \`\`\` Then build a DeckGL Map for it - no layers just yet! \`\`\`js tripdeckgl = new deck.DeckGL({ container: tripcontainer, map: mapboxgl, mapboxAccessToken: '', mapboxApiAccessToken: token, mapStyle: 'mapbox://styles/mapbox/dark-v10', layers: [], initialViewState: { longitude: -82.9835, latitude: 42.2767, zoom: 11.5, minZoom: 3, maxZoom: 15, pitch: 0 }, controller: true }); \`\`\` `37:[{"id":"f90c064fc7fc4eccc07748ee71cf04eadd6af01ea3b46e94d04185f78ebac28429e8c6a5c32c2c2e0f049f61c0f4e48cedbfc49b971fccf5b061cd41c7d175cc","href":"https://static.observableusercontent.com/files/f90c064fc7fc4eccc07748ee71cf04eadd6af01ea3b46e94d04185f78ebac28429e8c6a5c32c2c2e0f049f61c0f4e48cedbfc49b971fccf5b061cd41c7d175cc","name":"snowplow-2021-02-15_2021-02-21_filtered.csv","type":"text/csv","lastModified":1615803992116.89,"size":8512637}] 2f:{"title":"Tutorial on DeckGL","cells":[{"id":0,"value":"md`# Tutorial on DeckGL`","pinned":false,"hidden":false,"mode":"ojs"},{"id":349,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":360,"value":"md`To start we'll need to import some dependancies. The first is mapbox - which DeckGL uses for tiling. The other dependancy is DeckGL itself.\n\\`\\`\\`js\nmapboxgl = require('mapbox-gl@^1.5.1/dist/mapbox-gl.js');\n\ndeck = require.alias({\n h3: {},\n s2Geometry: {}\n})('deck.gl@8.4.1/dist.min.js')\n\\`\\`\\`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":602,"value":"md``","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"mapboxgl = require('mapbox-gl@^1.5.1/dist/mapbox-gl.js')","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"deck = require.alias({\n h3: {},\n s2Geometry: {}\n})('deck.gl@8.4.1/dist.min.js')","pinned":false,"hidden":false,"mode":"ojs"},{"id":605,"value":"md``","pinned":false,"hidden":false,"mode":"ojs"},{"id":394,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":18,"value":"getDeckGLMap = (layers, pitch) => {\n if (pitch === undefined) {\n pitch = 0;\n }\n\n let container = html`

`;\n\n let deckgl = new deck.DeckGL({\n container,\n map: mapboxgl,\n mapboxAccessToken: '',\n mapboxApiAccessToken: token,\n mapStyle: 'mapbox://styles/mapbox/dark-v10',\n layers: layers,\n initialViewState: {\n longitude: -82.9835,\n latitude: 42.2767,\n zoom: 11.5,\n pitch: pitch\n },\n controller: true\n });\n\n return container;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":427,"value":"md`Heres what the map looks like if we pass in no layers using this code: \n\\`\\`\\`javascript\ngetDeckGLMap([])\n\\`\\`\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"getDeckGLMap([])","pinned":false,"hidden":false,"mode":"ojs"},{"id":432,"value":"md`Great! Now we have a map. Not very interesting though - we should put things on the map. Before we can do that we need some data. I have attached two mid-February snowy Windsor days to this workbook. We import the data using this code:\n\n\\`\\`\\`js\ndata = FileAttachment(\"snowplow-2021-02-15_2021-02-21_filtered.csv\").csv(\n {typed: true})\n\\`\\`\\`\nFileAttachment references an attached file to the workbook - it's an Observable API call. The csv function call is another Observable feature that parses CSV and gives you an array of javascript objects - very helpful. The typed parameter tells the CSV parser to try and assign data types to the data when it parses the CSV text.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":27,"value":"data = (await FileAttachment(\"snowplow-2021-02-15_2021-02-21_filtered.csv\").csv({\n typed: true\n})).filter(\n d =>\n d[\"Plow status\"] === \"TRUE\" ||\n d[\"Spreader status\"] === \"TRUE\" ||\n d[\"Engine status\"] === \"TRUE\"\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":443,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":34,"value":"getDeckGLMap([\n new deck.ScatterplotLayer({\n id: 'scatterplot-layer',\n data,\n pickable: true,\n opacity: 1,\n filled: true,\n radiusScale: 1,\n radiusMinPixels: 1,\n radiusMaxPixels: 1,\n getPosition: d => [d.Longitude, d.Latitude],\n getRadius: d => 1,\n getFillColor: d => [255, 140, 0]\n })\n])","pinned":false,"hidden":false,"mode":"ojs"},{"id":474,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":55,"value":"getDeckGLMap([\n new deck.HeatmapLayer({\n id: 'heatmapLayer',\n data,\n getPosition: d => [d.Longitude, d.Latitude],\n aggregation: 'COUNT',\n radiusPixels: 10,\n colorDomain: [100, 1000]\n })\n])","pinned":false,"hidden":false,"mode":"ojs"},{"id":484,"value":"md`\nIn general, if you are mapping a lot of points and want and idea of distribution, another good option is hex maps. Hex maps divide the map into hexagons. We can then count the points inside the hexagons.\n\\`\\`\\`javascript\ngetDeckGLMap(\n [\n new deck.HexagonLayer({\n id: 'hexagon-layer',\n data,\n pickable: true,\n extruded: true,\n radius: 50,\n elevationScale: 4,\n colorDomain: [1, 150],\n elevationDomain: [1, 150],\n elevationRange: [0, 500],\n getPosition: d => [d.Longitude, d.Latitude]\n })\n ],\n 40\n)\n\\`\\`\\`\nThe definition looks a lot like the heatmap except I've also made this map extruded, added some elevation parameters, and set the pitch at 40. That's right! DeckGL also allows you to build 3D map visualizations. Here the extruded hexagons represent the same value as the color - the number of points. But, hex maps can also show another dimension for the data if you had another data point that you'd like to display.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":97,"value":"getDeckGLMap(\n [\n new deck.HexagonLayer({\n id: 'hexagon-layer',\n data,\n pickable: true,\n extruded: true,\n radius: 50,\n elevationScale: 4,\n colorDomain: [1, 150],\n elevationDomain: [1, 150],\n elevationRange: [0, 500],\n getPosition: d => [d.Longitude, d.Latitude]\n })\n ],\n 40\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":496,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"d3 = require(\"d3@6\");","pinned":false,"hidden":false,"mode":"ojs"},{"id":145,"value":"getColorScale = (min, max) => {\n let func = d3\n .scaleSequential()\n .domain([min, max])\n .interpolator(d3.interpolateRainbow);\n \n return d => {\n let col = d3.color(func(d));\n return [col.r, col.g, col.b];\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":129,"value":"truckpaths = {\n let reduced = data.reduce((accum, d) => {\n if (accum[d.Truck] === undefined) {\n accum[d.Truck] = {\n Truck: d.Truck,\n path: [],\n timestamps: []\n };\n }\n accum[d.Truck].path.push([d.Longitude, d.Latitude]);\n accum[d.Truck].timestamps.push(new Date(d[\"Date time\"]).getTime());\n return accum;\n }, {});\n\n let colorScale = getColorScale(0, Object.keys(reduced).length);\n let trucks = [];\n let num = 0;\n for (let truckid in reduced) {\n reduced[truckid].color = colorScale(num);\n trucks.push(reduced[truckid]);\n num++;\n }\n\n return trucks;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":518,"value":"md``","pinned":false,"hidden":false,"mode":"ojs"},{"id":513,"value":"md `That was a long walk! But, now we can create a [Path layer](https://deck.gl/docs/api-reference/layers/path-layer) showing all the truck paths!\n\n\\`\\`\\`js\ngetDeckGLMap(\n [\n new deck.PathLayer({\n id: 'path-layer',\n data: truckpaths,\n pickable: true,\n widthScale: 1,\n widthMinPixels: 1,\n getPath: d => d.path,\n getColor: d => d.color,\n getWidth: d => 1\n })\n ]\n)\n\\`\\`\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":173,"value":"getDeckGLMap(\n [\n new deck.PathLayer({\n id: 'path-layer',\n data: truckpaths,\n pickable: true,\n widthScale: 1,\n widthMinPixels: 1,\n getPath: d => d.path,\n getColor: d => d.color,\n getWidth: d => 1\n })\n ]\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":528,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":271,"value":"tripdeckgl = new deck.DeckGL({\n container: tripcontainer,\n map: mapboxgl,\n mapboxAccessToken: '',\n mapboxApiAccessToken: token,\n mapStyle: 'mapbox://styles/mapbox/dark-v10',\n layers: [],\n initialViewState: {\n longitude: -82.9835,\n latitude: 42.2767,\n zoom: 11.5,\n minZoom: 3,\n maxZoom: 15,\n pitch: 0\n },\n controller: true\n });","pinned":false,"hidden":false,"mode":"ojs"},{"id":556,"value":"md `Next, calculate the minimum timestamp and create execution block in a cell that creates a trips layer. Instead of assinging the layer when we create the DeckGL map, we are assigning it using the setProps call whenever a variable inside this code block changes.\n\nTo help navigation, we minus the first timestamp we have from the timestamps in our dataset. That way our currentTime can be zero based. It also helps prevent potential overflows.\n\n\\`\\`\\`js\nminTimestamp = d3.min(data, d => new Date(d[\"Date time\"]).getTime())\n\n{\n let layer = new deck.TripsLayer({\n id: 'trips-layer',\n data: truckpaths,\n getPath: d => d.path,\n getTimestamps: d => d.timestamps.map(p => p - minTimestamp),\n getColor: d => d.color,\n opacity: 0.8,\n widthMinPixels: 3,\n rounded: true,\n trailLength: 2000000,\n currentTime: currentMinute * 60 * 1000\n })\n \n tripdeckgl.setProps({\n layers: [\n layer\n ]\n })\n\n return \"onchange\"\n}\n\n\\`\\`\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":210,"value":"minTimestamp = d3.min(data, d => new Date(d[\"Date time\"]).getTime())","pinned":false,"hidden":false,"mode":"ojs"},{"id":275,"value":"{\n\n let layer = new deck.TripsLayer({\n id: 'trips-layer',\n data: truckpaths,\n getPath: d => d.path,\n getTimestamps: d => d.timestamps.map(p => p - minTimestamp),\n getColor: d => d.color,\n opacity: 0.8,\n widthMinPixels: 3,\n rounded: true,\n trailLength: 2000000,\n currentTime: currentMinute * 60 * 1000\n })\n \n tripdeckgl.setProps({\n layers: [\n layer\n ]\n })\n\n return \"onchange\"\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":563,"value":"md `Now we need a parameter to drive changes to the currentTime variable in our code block. We import a Scrubber from another notebook, and create a value for that scrubber for each minute in our dataset.\n\n\\`\\`\\`js\nimport {Scrubber} from \"@noahthemeh/scrubber\"\n\nmaxAdjustedTimestamp = d3.max(data, d => new Date(d[\"Date time\"]).getTime()) - minTimestamp\nmaxMinutes = Math.floor(maxAdjustedTimestamp /1000 / 60)\nminutes = Array.from({length: maxMinutes}, (_, i) => i)\n\npad = (d) => {\n if(d < 10){\n return \"0\" + d;\n }\n else{\n return d;\n }\n}\n\nviewof currentMinute = Scrubber(minutes, \n{\n delay: 100,\n format: d => {\n let date = new Date(minTimestamp + d * 60 * 1000);\n return date.getFullYear() + \"-\" + pad(date.getMonth() + 1) + \"-\" + pad(date.getDate()) + \" \" + pad(date.getHours()) + \":\" + pad(date.getMinutes()) \n }\n})\n\\`\\`\\`\nviewof above is a special Observable notebook indicator showing that changes to the HTML control element in question should change the value of its cell.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":297,"value":"import {Scrubber} from \"@noahthemeh/scrubber\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":283,"value":"maxAdjustedTimestamp = d3.max(data, d => new Date(d[\"Date time\"]).getTime()) - minTimestamp","pinned":false,"hidden":false,"mode":"ojs"},{"id":534,"value":"maxMinutes = Math.floor(maxAdjustedTimestamp /1000 / 60)","pinned":false,"hidden":false,"mode":"ojs"},{"id":285,"value":"minutes = Array.from({length: maxMinutes}, (_, i) => i)","pinned":false,"hidden":false,"mode":"ojs"},{"id":339,"value":"pad = (d) => {\n if(d < 10){\n return \"0\" + d;\n }\n else{\n return d;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":581,"value":"md `Here it is! The map with a scrubber!`","pinned":false,"hidden":false,"mode":"ojs"},{"id":294,"value":"viewof currentMinute = Scrubber(minutes, {\n delay: 100,\n autoplay: true,\n loop: true,\n format: d => {\n let date = new Date(minTimestamp + d * 60 * 1000);\n return (\n date.getFullYear() +\n \"-\" +\n pad(date.getMonth() + 1) +\n \"-\" +\n pad(date.getDate()) +\n \" \" +\n pad(date.getHours()) +\n \":\" +\n pad(date.getMinutes())\n );\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":269,"value":"tripcontainer = html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":585,"value":"md``","pinned":false,"hidden":false,"mode":"ojs"},{"id":586,"value":"md``","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"token = {\n // Don't use this token. It will be rotated frequently. Get your own at https://mapbox.com\n return 'pk.eyJ1Ijoiam9obmhhbGRlbWFuIiwiYSI6ImNrMjIxNnBrdTAxN3Mzb3BmbWFzbnV6OGoifQ.HrfAiASrw9EQetZn13xAPg';\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W37","stdlib":"1","titling":"auto"}