31:Td13,md`# Notes ## On Data Note--some changes on 11-17 so this can stay up to date. Now I use Hopkins instead of the New York times for data. The data here--[from the Johns Hopkins COVID tracking project](https://github.com/nytimes/covid-19-data/blob/master/us-counties.csv)--is imperfect in both general ways (case counts increase as testing goes up, so there were--for instance--certainly far more cases in New York City in early March than are shown here) and local ones (states sometimes dump a bunch of counts into the stats at once; this seems to be doing weird things to Georgia at various points.) Earlier versions of this map online assign all counts to "New York City" in Manhattan--here there seem to be somewhat separate borough-by-borough counts. In general, Hopkins reports *cumulative* counts; I treat each count as coming from the first day that it appears in the reports. Sometimes case numbers go down; I just throw those days away, on the theory that the errors those shifts capture are probably distributed across earlier weeks in ways that aren't easy to understand. Daily COVID counts are kind of a mess, because some jurisdictions don't count over the weekend; the data I use here is a sum of cases over the previous fourteen days. Rate of change is calculated as the ratio between the seven days preceding any county's counts, and the 7-14 days before them. (I.e. "last weeks cases" divided by "the week before that's cases.") Because it's unpleasant to watch the states change size up and down, I then apply some smoothing on top of those counts before rendering. I use a three day moving average several times, on the idea that it approximates using a Gaussian kernel smoothing. This was costly in the library I was using before, so now it happens straight on some Array buffers. ## On visualization This is using Regl and triangulated shapes using Earcut. I [store the triangulations in their binary form using Apache Arrow as a static format](https://observablehq.com/@bmschmidt/a-binary-file-format-for-projected-triangulated-shapefile). For the technical people out there, that's probably the most interesting thing here; I would have no idea how to do something like this in GIS software, but once you're dealing with thousands of little triangles, it's easy to distort them with basic trig functions, calculate areas, and apply other data visualization tools. And WebGL is so fast that we can serve unsimplified polygons--here the Census bureau's best, 500k resolution county shapefiles, which matters because distortion means we might actual see the islands in Miami, the Rockaways in Queens, or the precise shape of Cook County even with the map at a natural resolution. (One thing that would be relatively straightforward would be randomly generating points *inside* the shapes here). I use the centroid of each polygon as the reference point; it would be possible to switch the anchor point to--say--the center of the metropolitan statistical area. No idea what that would look like! A few other notes. * Scaling by area is approximate with distortions; I calculate the desired size based on undistorted area, and there's no guarantee that the area of a county stays the same through all the distortions. * There's a snap each day as the sizes go up or down. It would be nice to smooth that away, too. `32:T6af,function text(date) { // Return a different caption depending on the scrubber state. if (date < "2020-03-18") return ` # Early March: Not our problem Real outbreaks in Seattle in New York City are mostly invisible due to a lack of testing; most Americans think this is a problem still confined to China, Iran, Italy, and Spain. ` if (date < "2020-05-01") return ` # March and April: First wave Huge localized outbreaks in New York City (here showing all cases as Manhattan), New Orleans, and Michigan, and elsewhere kill tens of thousands. Testing rates are still low enough that these numbers probably mislead. Most of the nation begins to lock down. Because the starting point is so low, much of the nation is churning with growth as the rate of change spikes up. By the end of the period, growth in New York subsides and the spike begins to decrease. ` if (date <= "2020-07-25") return ` # First reprieve The crisis in New York subsides, and the rest of the country receives some reprieve. ` if (date > "2020-10-15") return ` # Spinning out of control Since October 15, it's just a nationwide clusterfuck. ` if (date > "2020-08-25") return ` # Second reprieve At the end of August, things got under control in the sun belt. ` if (date > "2020-07-25") return ` # The Sunbelt wave This is probably where we lost. Rather than regroup in the summer and stamp out the virus into local, containable clusters, things spike massively out of control. The president holds a rally that kills former presidential candidates of his own party; bikers converge on South Dakota; kids start returning to college and attending parties even after they get positive tests. ` return "No text" }33:Tbcc,draw_frame = function(date) { const overall_radius = jitter_radius / 100 * (Math.sin(Date.now()/1000) + 1)/2 const prop_list = [] let ix = -1 for (let feature of hi_res_counties) { ix += 1 if (feature.vertices === null) {continue} if (feature.coordinates === null) {continue} if (handler.element_handler.get(ix) === undefined) { handler.element_handler.set(ix, gl.elements({ primitive: 'triangles', usage: 'static', data: feature.vertices, type: "uint" + feature.coord_resolution, length: feature.vertices.length, // in bytes count: feature.vertices.length / feature.coord_resolution * 8 })) handler.coord_handler.post_data(ix, feature.coordinates) } if (handler.get_prop("angle", ix) === undefined) handler.set_prop("angle", ix, 0) let rand = handler.get_prop("rand", ix) || handler.set_prop("random", ix, 0) const { STATE, COUNTY } = feature let c_stats = long_term_corona_counts.get(STATE + COUNTY + "-" + date) const pop = pops.get(`${STATE}${COUNTY}`) if (c_stats == undefined) { c_stats = {delta: 1, rolling: 0}; } let { delta, rolling } = c_stats; const rate = rolling/pop // Increment the current angle by an amount. const pop_per_area = pop/(feature.CENSUSAREA + 1e-10) const magnitude = distortion(rate) if (isNaN(delta)) { delta = 1 } else if (delta <= 0) { delta = .1 } handler.set_prop("angle", ix, handler.get_prop("angle", ix) + delta_scale(delta)) const t = Math.sin(Date.now()/(600+rand[1]*10)) let color = incidence_color(rate) || 'rgb(0, 0, 0)' let [r, g, b] = color.split(/$|$/)[1].split(",").map(d => +d) //r = 128; g = 128; b = 128; let x_shift = -feature.centroid_x let y_shift = -feature.centroid_y const v = { color: [r/255, g/255, b/255], zoom, centroid: [feature.centroid_x, feature.centroid_y], angle: handler.get_prop("angle", ix), scale: scale_by_votes == "yes" ? size_scale(rolling/feature.CENSUSAREA) : 1, elements: handler.element_handler.get(ix), coords: handler.coord_handler.get(ix), translate: [0, 0], incidence: incidence } prop_list.push(v) } render(prop_list) } 34:T773,renderer = ( //Starting point: https://observablehq.com/@marcom13/mesh-rendering-using-webgl-regl { vert: ` precision mediump float; attribute vec2 position; uniform float aspect; uniform float u_zoom; uniform float u_scale; uniform float u_incidence; uniform float u_theta; uniform vec3 color; uniform vec2 translate; uniform vec2 u_centroid; varying vec3 fragColor; mat2 rotate2d(float _angle){ return mat2(cos(_angle),-sin(_angle), sin(_angle),cos(_angle)); } void main () { gl_PointSize = 2.; vec2 rot; fragColor = color; vec2 p_from_centroid = position - u_centroid; if (u_incidence == 0.) { rot=p_from_centroid*rotate2d(u_theta); } else { float angle = atan(p_from_centroid.x, p_from_centroid.y); float magnitude = sqrt(dot(p_from_centroid, p_from_centroid)); float adjust = magnitude * (2. + sin(angle - u_theta)) / 2.; if (u_scale != 1. && adjust > magnitude) { adjust = magnitude; } rot = vec2(adjust * sin(angle), adjust * cos(angle)); //rot = vec2(.03 * sin(angle), .03 * cos(angle)); } vec2 rot_adj = rot * u_scale + u_centroid; gl_Position = vec4( rot_adj.x *u_zoom + translate.x * u_zoom, -rot_adj.y * u_zoom * aspect - translate.y*u_zoom*aspect, 0., 1.); } `, frag: ` precision mediump float; varying vec3 fragColor; void main () { gl_FragColor = vec4(fragColor, 1.); } `, attributes: { position: (state, props) => props.coords }, elements: function (state, props) {return props.elements}, uniforms: { u_centroid: gl.prop('centroid'), u_theta: (_, {angle}) => angle !== undefined ? angle : 0, u_zoom: (_, {zoom}) => zoom * zoom, u_scale: (_, {scale}) => scale, u_incidence: (_, {incidence}) => incidence == "distortion" ? 1 : 0, aspect: width/height, translate: (state, props) => props.translate, color: (state, props) => props.color }, primitive: "triangle", } )35:T6d9,{ return "in process-merging multiple triangulated features together." const g = {} for (let f of ["coordinates", "vertices", "coord_resolution", "pixel_area", "centroid_x", "centroid_y"]) { g[f] = [hi_res_counties.getColumn(f).get(1), hi_res_counties.getColumn(f).get(2), hi_res_counties.getColumn(f).get(16)] } const l = d3.sum(g.coordinates.map(d => d.length)) const coordinates = new Uint8Array(l) let offset = 0 for (let element of g.coordinates) { coordinates.set(element, offset) offset += element.length } const n_points = coordinates.length * 4 / 2; // (i.e.: 4 byte floats, two floats per point). const coord_resolution = n_points < 2**8 ? 8 : n_points < 2**16 ? 16 : 32 const lengths = g.vertices.map((d, i) => d.length/g.coord_resolution[i] * 8) const Buffer = getInt(coord_resolution) const vertices = new Buffer(d3.sum(lengths)) let buffer_offset = 0; let index_offset = 0; g.vertices.forEach((d, i) => { const Res = getInt(g.coord_resolution[i]) const indices = new Res(g.vertices[i].buffer) // PROBLEM--ACCESSING THE BUFFER THIS WAY RETURNS THE WHOLE UNDERLYING BUFFER, NOT JUST THE SLICE // REPRESENTING THE DATA WE'RE INTERESTING IN. NEED TO INCLUDE OFFSETS SOMEHOW. console.log(indices, g.coord_resolution[i]) indices.forEach((index, i) => { vertices[buffer_offset] = indices[i] + index_offset buffer_offset += 1 }) index_offset += lengths[i] }) const pixel_area = d3.sum(g.pixel_area) const centroid_x = d3.sum(d3.zip(g.centroid_x, g.pixel_area).map(([a, b]) => a * b/pixel_area)) const centroid_y = d3.sum(d3.zip(g.centroid_y, g.pixel_area).map(([a, b]) => a * b/pixel_area)) return { coordinates, vertices, coord_resolution, pixel_area, centroid_x, centroid_y} }36:T46a,long_term_corona_counts = { const counts = new Float32Array(date_names.length) const rolling = new Float32Array(date_names.length) const delta = new Float32Array(date_names.length) const datecounts = new Map() for (let row of t2) { // return row const val = row.values counts[0] = val[0] for (let i = 1; i < val.length; i++) { counts[i] = val[i] > val[i-1] ? val[i] - val[i-1] : 0 } for (let i = 0; i < 14; i++) { rolling[i] = d3.sum(counts.slice(0, i)) } for (let i = 14; i < counts.length; i ++) { rolling[i] = d3.sum(counts.slice(i-7, i)) } delta.fill(0) // NB this must be even or else counts and delta have to be switched. blur(rolling, delta, 4) for (let i=0; i < rolling.length; i++) { if (i < 7) {delta[i] = 1} else { delta[i] = rolling[i]/rolling[i-7] } } for (let i = 0; i < delta.length; i++) { const date = date_names[i] datecounts.set(`${row.FIPS}-${date_names[i]}`, {rolling: rolling[i], delta: delta[i]}) } } return datecounts }37:[{"id":"9480301ff3573185a91b55c0c464ecaa34ca0f1bdbb3c4210fe66a60e3fea3f256a080a27386a5809f6af953a442bd89aef9bc6ce56e2e7ab595befd93067814","href":"https://static.observableusercontent.com/files/9480301ff3573185a91b55c0c464ecaa34ca0f1bdbb3c4210fe66a60e3fea3f256a080a27386a5809f6af953a442bd89aef9bc6ce56e2e7ab595befd93067814","name":"full@2.feather","type":"application/octet-stream","lastModified":1605280013582.211,"size":13689346},{"id":"f2f1518e9158c3a66854646a10b0a83bc124e5055b2ee7bd0d1af3e322118488f49a273eb03ec32ba60b511d63cf2bfda7a4e19c5084af62072065ed4bc01bcf","href":"https://static.observableusercontent.com/files/f2f1518e9158c3a66854646a10b0a83bc124e5055b2ee7bd0d1af3e322118488f49a273eb03ec32ba60b511d63cf2bfda7a4e19c5084af62072065ed4bc01bcf","name":"countries (3).gleofeather","type":"application/octet-stream","lastModified":1605464785811.222,"size":11462768}] 2f:{"title":"Spinning out of control","cells":[{"id":0,"value":"md`# Spinning out of control\n\nCan rotation be a useful animated encoding channel on maps? [Using regl and earcut, it's possible to smoothly animate several thousand counties even with a high level of zoom.](https://observablehq.com/@bmschmidt/regl-and-earcut-for-rendering-very-detailed-polygons) On the principle that anything possible must be good, let's try a quick prototype where:\n\n* **Colors show cases of Coronavirus per capita,**\n* **Size shows number of cases in the previous 14 days.** (Smoothed).\n* while **spin shows *rate of change over the previous 7 days.*** (Of smoothed values)\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2278,"value":"md`## The US Pandemic on **${ddate.toUTCString().slice(4, 17)}**`","pinned":false,"hidden":false,"mode":"ojs"},{"id":200,"value":"myCanvas = DOM.canvas(width, height)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1548,"value":"viewof ddate = Scrubber(dates.slice(40,1000), {\n autoplay: true,\n delay: 720,\n loop: false,\n loopDelay: 20000,\n initial: 30,\n title: \"Date\",\n width,\n format: date => date.toLocaleString(\"en\", {month: \"long\", day: \"numeric\"})\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1603,"value":"md`\n\n${text(date)}\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1328,"value":"viewof incidence = radio({description: \"One kind of interesting option is to use not spin but angular distortion--this makes areas seethe uneasily. \", title: \"Spin type\", options: [\"distortion\", \"spin\"], value: \"distortion\"})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1250,"value":"viewof scale_by_votes = radio({title: \"Scale by cases?\", options: [\"yes\", \"no\"], value: \"yes\", description: \"\"}) // Variable name left over from an old notebook.","pinned":false,"hidden":false,"mode":"ojs"},{"id":1748,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":1700,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":1590,"value":"draw_loop = {while(true) {\n yield(draw_frame(date))\n}}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1310,"value":"dates = date_names.map(d => new Date(Date.parse(d)))","pinned":true,"hidden":false,"mode":"ojs"},{"id":967,"value":"viewof zoom = slider({title: \"zoom (square root)\", min: .5, max: 15, value: 1, step: .1})","pinned":true,"hidden":false,"mode":"ojs"},{"id":537,"value":"viewof jitter_radius = slider({'title': \"jitter radius\", 'min': 0, 'max': 25, value: 1, step: .1, \"description\": \"Jitter makes the counties wobble, and the visualization even more unpleasant.\"})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1178,"value":"viewof speed = slider({title: \"Speed of spin adjustment\", min: -5, max: 5})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1824,"value":"md`## Some scales\n\nNot sure if I still use them all!\n`","pinned":true,"hidden":false,"mode":"ojs"},{"id":1450,"value":"rate_of_change = d3.scaleDivergingLog().domain([5,1, .2]).interpolator(d3.interpolatePiYG).clamp(true)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1644,"value":"incidence_color = d3.scaleDivergingLog().domain([1e-2, 1e-3,1e-4 ]).interpolator(d3.interpolateRdBu).clamp(true)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1555,"value":"date = ddate.toISOString().slice(0, 10)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1517,"value":"distortion = d3.scaleSqrt().domain([0, 1/1000]).range([0, 1])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1570,"value":"size_scale = d3.scaleSqrt().domain([0, 50]).range([0, 10])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1657,"value":"delta_scale = d3.scaleDivergingLog().domain([.2, 1, 10]).clamp(true).range([0, 0, 1])","pinned":true,"hidden":false,"mode":"ojs"},{"id":1826,"value":"md`## Draw command`","pinned":true,"hidden":false,"mode":"ojs"},{"id":1890,"value":"hi_res_counties.get(0).vertices","pinned":true,"hidden":false,"mode":"ojs"},{"id":1586,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":1861,"value":"import { ReglBufferHandler } from '@bmschmidt/pre-triangulated-binary-maps-for-webgl'","pinned":true,"hidden":false,"mode":"ojs"},{"id":1868,"value":"handler = new ReglBufferHandler(gl)","pinned":true,"hidden":false,"mode":"ojs"},{"id":209,"value":"gl = regl({canvas:myCanvas});","pinned":true,"hidden":false,"mode":"ojs"},{"id":1564,"value":"start_time = Date.now()\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":213,"value":"import { radio, slider } from '@jashkenas/inputs'","pinned":true,"hidden":false,"mode":"ojs"},{"id":222,"value":"render = gl(renderer);","pinned":true,"hidden":false,"mode":"ojs"},{"id":163,"value":"colorscheme = d3.scaleLinear().domain([0, 100]).range(d3.schemeBuGn)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2149,"value":"groups = {return {36005: \"New York\", 36047: \"New York\", 36061: \"New York\", 36081: \"New York\", 36085: \"New York\"}}","pinned":true,"hidden":false,"mode":"ojs"},{"id":726,"value":"function tri_merge(triangles, coords, int_sizes) {\n \n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2154,"value":"feathered","pinned":true,"hidden":false,"mode":"ojs"},{"id":45,"value":"$34","pinned":true,"hidden":false,"mode":"ojs"},{"id":1887,"value":"height = width * .75","pinned":true,"hidden":false,"mode":"ojs"},{"id":2185,"value":"$35","pinned":true,"hidden":false,"mode":"ojs"},{"id":2202,"value":"function getInt(n) {\n if (n == 8) {return Uint8Array}\n if (n == 16) {return Uint16Array}\n if (n == 32) {return Uint32Array}\n throw \"No such buffer size\"\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2221,"value":"new Uint32Array((new Uint8Array([1, 2, 3, 4]).buffer))","pinned":true,"hidden":false,"mode":"ojs"},{"id":805,"value":"hi_res_counties.schema.fields.map(d => d.name)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1527,"value":"import {Scrubber} from \"@mbostock/scrubber\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1333,"value":"arrow = require(\"apache-arrow\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1357,"value":"raw_data = FileAttachment(\"full@2.feather\").arrayBuffer()","pinned":true,"hidden":false,"mode":"ojs"},{"id":1336,"value":"feathered = arrow.Table.from(raw_data)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1350,"value":"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1855,"value":"hi_res_counties = arrow.Table.from(await FileAttachment(\"countries (3).gleofeather\").arrayBuffer())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1859,"value":"hi_res_counties.schema","pinned":true,"hidden":false,"mode":"ojs"},{"id":1447,"value":"import { pops as raw_pops } from '@codingwithfire/cmu-covidcast-api-bubbles-export'","pinned":true,"hidden":false,"mode":"ojs"},{"id":1947,"value":"pops = {\n // raw_pops.set(\"36061\", 8.5e06)\n return raw_pops\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1348,"value":"import {aq, op} from '@uwdata/arquero'","pinned":true,"hidden":false,"mode":"ojs"},{"id":63,"value":"regl = require(\"regl\") // Use the latest Version","pinned":true,"hidden":false,"mode":"ojs"},{"id":1982,"value":"hopkins = aq.fromCSV(await fetch(\"https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_US.csv\").then(d => d.text()))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1988,"value":"t2 = hopkins\n .select(aq.not([\"Lat\", \"Long\", \"Long_\", \"code3\", \"iso3\", \"iso2\", \"UID\", \"Admin2\", \"Province_State\", \"Country_Region\"]))\n .derive({FIPS: d => op.padstart(d.FIPS, 5, \"0\")})\n .fold(aq.range(2, hopkins.numCols() - 1))\n .groupby([\"FIPS\", \"Combined_Key\"])\n .orderby(\"key\")\n .rollup({values: d => op.values(d.value)})\n\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":2018,"value":"date_names = hopkins.columnNames().map(d => [d, new Date(Date.parse(d))]).filter(d => Date.parse(d[0])).map(d => d[1].toISOString().slice(0, 10))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2014,"value":"$36","pinned":true,"hidden":false,"mode":"ojs"},{"id":2041,"value":"function blur(ping, pong, passes=1) {\n if (passes == 0) {\n return \n }\n for (let i=1; i < ping.length-1; i++) {\n pong[i] = (ping[i-1] + ping[i] + ping[i+1])/3\n }\n pong[0] = (ping[0] + ping[1])/2\n pong[pong.length-1] = (ping[pong.length-1] + ping[pong.length-2])/2\n blur(pong, ping, passes-1)\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":9,"value":"d3 = require(\"d3@v6\", \"d3-geo-projection\")","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W37","stdlib":"1","titling":"auto"}