31:T507,md`### NY State Scaling (~20 minutes) To allow the chart to extend upwards (beyond the bounds of the state), the **range** of the scale needs to be adjusted. It turns out that this is tricky, based on how bar charts are often implemented using D3 (including how Aaron had done it): ~~~js // y scale: a value of 0 returns the maximum height, and the highest data value returns 0 y_scale = d3 .scaleLinear() .domain([0, max_value]) .range([state_size, 0]) // Then, when drawing your bars // Move down a certain number of pixels from the top, based on the current value .attr('y', d => y(d.value)) // Set the height as the smallest value minus the amount moved down from the top .attr('height', d => y(0) - y(d.value)) ~~~ Alternatively, we could do the following (but it requires switching the range) ~~~js // y scale: a value of 0 returns the minimum height, and the highest data value the max height y_scale = d3 .scaleLinear() .domain([0, max_value]) .range([0, state_size + 100]) // switched from above, allow extra pixels above // Then, when drawing your bars // Move down the size of the state, minus the value for this rectangle .attr('y', d => state_size - y(d.value)) // Draw the rectangle height based on the current value .attr('height', d => y(d.value)) ~~~ `32:T48b,md`## Refactoring Code (~1.5 hours) Refactoring is the process of reorganizing code into different components so that the code is more efficient, expressive, or modular. While Aaron's work was a great starting point, by the end, there were many pieces that didn't make sense any more, and the solution I had implemented was quite repetitive (lots of copying and pasting, for example, to render the legend). So, I started to reorgainze the [code](https://observablehq.com/@mkfreeman/small-multiple-chart-cartogram/2) I had forked (not to make the outcome look any different, but to make it more usable for other people and other projects). Notable changes included: - Removing old data/code no longer reference (~20 minutes) - Reformatting the data to work with the (updated) scales that we had defined here (now also defined in that script) (~20 minutes) - Removing redundant code regarding the legend -- this was done by creating [this function](https://observablehq.com/@mkfreeman/small-multiple-chart-cartogram/2#renderState)(~40 minutes) And of course, another ~30 minutes of brushing up the writing here to make it a little smoother. Thanks for reading! `33:[{"id":"de38d39ff5fcd7f428925428d8d9baf2402b640eeca2236314573df36a876aef16d7bbe0a98c10b71eb4e8bdd2734412820d71ec488b428c9becb1ec6a5dd20b","href":"https://static.observableusercontent.com/files/de38d39ff5fcd7f428925428d8d9baf2402b640eeca2236314573df36a876aef16d7bbe0a98c10b71eb4e8bdd2734412820d71ec488b428c9becb1ec6a5dd20b","name":"hopkins_map.png","type":"image/png","lastModified":1584848281961.211,"size":458516},{"id":"03f3ca01e2de0c1593d32448a345256ba4e605965498308636f6ff797fefe338aca83f08a10c76438db7e79d8e2bf85d795c92bd12e78b25570a9b641ea5a635","href":"https://static.observableusercontent.com/files/03f3ca01e2de0c1593d32448a345256ba4e605965498308636f6ff797fefe338aca83f08a10c76438db7e79d8e2bf85d795c92bd12e78b25570a9b641ea5a635","name":"data_structure.png","type":"image/png","lastModified":1584850428292.831,"size":261975},{"id":"8d7dee7d56db87ea5191ab32f7806209b0b664ef4f7c22989908608eb7dc91edff7f1f46581fd2ab0f7200b197c13a105305aadf25387914eef0d7c1b68cc548","href":"https://static.observableusercontent.com/files/8d7dee7d56db87ea5191ab32f7806209b0b664ef4f7c22989908608eb7dc91edff7f1f46581fd2ab0f7200b197c13a105305aadf25387914eef0d7c1b68cc548","name":"data_structure_labeled.png","type":"image/png","lastModified":1584851860167.904,"size":266511},{"id":"41fab4c0dff6b6ff32f378a0eb04c9b73de7b6d3fc900d90b49155378e643f454f6c2ccea429ef7b01cb71671eaa80fe64b428598e8c59000b5f4b5fdb324def","href":"https://static.observableusercontent.com/files/41fab4c0dff6b6ff32f378a0eb04c9b73de7b6d3fc900d90b49155378e643f454f6c2ccea429ef7b01cb71671eaa80fe64b428598e8c59000b5f4b5fdb324def","name":"adjusted_scale.png","type":"image/png","lastModified":1584911646600.21,"size":92704},{"id":"caa89a10f139e05e8d8abf26552fd2123843d4a70d435c00cc83f1986237bff5c7a0c123ea64fc422e2d58ded7ecd90fd044d9a758d4b010508f3429a2daad66","href":"https://static.observableusercontent.com/files/caa89a10f139e05e8d8abf26552fd2123843d4a70d435c00cc83f1986237bff5c7a0c123ea64fc422e2d58ded7ecd90fd044d9a758d4b010508f3429a2daad66","name":"basic_cartogram.png","type":"image/png","lastModified":1584933094240.136,"size":91029},{"id":"2fea42020fedd15470788bf22c4749513e57ae80f9a127ba8a91e0290b1c6b46711af7dc6306358a13b99108b67af7974374fb4d5dc885444fcc3fe803f8872b","href":"https://static.observableusercontent.com/files/2fea42020fedd15470788bf22c4749513e57ae80f9a127ba8a91e0290b1c6b46711af7dc6306358a13b99108b67af7974374fb4d5dc885444fcc3fe803f8872b","name":"with_background_rects.png","type":"image/png","lastModified":1584933215634.047,"size":116810},{"id":"73c5b04468025a0b00fbffc02a0b3ef7caec734fce7df34378c28968b800995e07b3e484c34b4c708ac22bbdaa10d4fea3ed37a1d73181c9f62fe0b220e2075d","href":"https://static.observableusercontent.com/files/73c5b04468025a0b00fbffc02a0b3ef7caec734fce7df34378c28968b800995e07b3e484c34b4c708ac22bbdaa10d4fea3ed37a1d73181c9f62fe0b220e2075d","name":"with_text_labels.png","type":"image/png","lastModified":1584933277302.057,"size":167354}] 2f:{"title":"Recreating NYT U.S. Cases Map","cells":[{"id":0,"value":"md`# Recreating NYT U.S. Cases Map`","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"md`Even as someone with a Master's Degree in Public Health, I've been hesitant to contribute to the volume of visualizations regarding COVID-19 (there's just so much noise out there). This notebook is an educational exercise in _re_-creating [this excellent piece](https://www.nytimes.com/interactive/2020/03/21/us/coronavirus-us-cases-spread.html) from the New York Times. Here is the final product:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":213,"value":"forked_chart","pinned":false,"hidden":false,"mode":"ojs"},{"id":612,"value":"md`I attempted to recreate this _before_ the NYT made their data public. To read more about that process, see [this notebook](https://observablehq.com/@mkfreeman/recreating-nyt-u-s-cases-map-pre-data-release).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"md`## Data Processing`","pinned":false,"hidden":false,"mode":"ojs"},{"id":617,"value":"md`Thankfully, the NYT made their data [public](https://github.com/nytimes/covid-19-data), though we'll have to load and format it for visualization`","pinned":false,"hidden":false,"mode":"ojs"},{"id":111,"value":"md`### Loading + Filtering Data (~20 minutes)`","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"md`Once identifying the data of interest, it needs to be **loaded** and **formatted** for use in JavaScript. So, first the loading:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":56,"value":"// Load the data with `d3.csv()`\nraw_data = d3.csv(\n \"https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv\"\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":622,"value":"md`Here's a preview of the data (which is already in a quite usable format):`","pinned":false,"hidden":false,"mode":"ojs"},{"id":62,"value":"render_table(raw_data.slice(0, 10))","pinned":false,"hidden":false,"mode":"ojs"},{"id":115,"value":"md`### Formatting Data (~30 Minutes)\nThe _\"right format\"_ for your data is tricky, and depends heavily on what you intend to _do_ with that data. A common approach I use is:\n\n_Match the data structure to an existing example of the output you want to create_\n\nLuckily enough, I had seen a really similar project by the incredible [Aaron Williams](aaron.williams@washpost.com) regarding Opioid overdoses. It used a [small multiple cartograms](https://observablehq.com/@aboutaaron/small-multiple-chart-cartogram) technique to show how the opiod epidemic is impacting each state: `","pinned":false,"hidden":false,"mode":"ojs"},{"id":408,"value":"aarons_chart","pinned":false,"hidden":false,"mode":"ojs"},{"id":411,"value":"md`And, the beautiful thing about Observable is that you can import directly from other peoples' notebooks -- the chart above is imported directly as follows:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":418,"value":"import { chart as aarons_chart } from \"@aboutaaron/small-multiple-chart-cartogram\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":415,"value":"md`Looking through his project, he had constructed an _array of objects_, where each object represented a state. Each object had a \\`data\\` property, which for each potential encoding, contained _another_ array of objects (each one corresponding to a value in a bar chart). This is much easier to see that to read through:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":127,"value":"displayImage(data_structure, 400)","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"md`While that is a bit to wrap one's mind around, it isn't a far cry from out current data structure. We'll just need to:\n- Create an array of objects (one for each state, as in our \\`cartogram_data\\`)\n- For each object, create a \\`data\\` property, containing an array of objects (one corresponding to each _date_ for each state)\n- keeping track of the _date of first case_ and the _total_:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":669,"value":"fix = md`**Update**: I realized that the data is the _total_ cases at each date (**not** the number of _new_ cases per day). So, this requires that the number of new cases each day is calculated (by subtracting the number of cases the previous day).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":636,"value":"// Take the array of objects from our cartogram, and add `data` to them\nformatted_data = statesData.map(d => {\n // Filter down the NYT data to the current state, then construct the data object for this state\n let state_data = raw_data.filter(dd => dd.state == d.state_full);\n\n const data = state_data.map((dd, i) => {\n // Calculate the *new* cases today by subracting the cases yesterday\n let new_cases = 0;\n if (i === 0) new_cases = +dd.cases;\n else new_cases = +dd.cases - +state_data[i - 1].cases;\n return { key: dd.date, value: new_cases };\n });\n d.state = d.state_full; // for naming consistency\n d.total = d3.sum(data, d => d.value);\n d.x_min = d3.min(data, dd => dd.key);\n\n // Format the date\n const date_pieces = d.x_min.split(\"-\");\n const x_min_label = parseDate(\n new Date(2020, date_pieces[1] - 1, date_pieces[2])\n );\n d.x_min_label = x_min_label;\n d.data = data;\n return d;\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":167,"value":"md`With the data in the right format, it _should_ be possible to pass it to the chart...`","pinned":false,"hidden":false,"mode":"ojs"},{"id":92,"value":"md`## Mapping (Using Aaron's Code)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"md`One of the most powerful things about Observable is the ability to **import** values from other notebooks (see more about imports [here](https://observablehq.com/@observablehq/introduction-to-imports)). Because Aaron's chart is (essentially) what we want to achieve, we should be able to simply import his chart, and specify our \\`formatted_data\\` as the data to use (_\"should\"_ being the operative word -- this step is always a little nervewracking...):\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":174,"value":"// Import his chart, but specify *our* data to use\nimport { chart } with {\n formatted_data as data\n} from \"@aboutaaron/small-multiple-chart-cartogram\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":183,"value":"md`Oh no! In trying to display the \\`chart\\` object, we get the message, RuntimeError: chart could not be resolved`","pinned":false,"hidden":false,"mode":"ojs"},{"id":191,"value":"md`So, **debugging**: there's a bit of a difference between Aaron's data structure and ours -- he anticipated switching between displayed values, so stored the data under a particular key:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":195,"value":"displayImage(data_structure_labeled, 400)","pinned":false,"hidden":false,"mode":"ojs"},{"id":201,"value":"md`At this point, we can either:\n- Fork his code and make change\n- Change our data structure to match his\n\nWhile his project is amazing, I'm anticipating that I might want to make changes to his code to achieve the layout of the NYT, so I just decided to make my own fork. In my fork, I changed references to the data above:\n\n~~~js\n// Originally was\n .data(d => d.data[selectedEncoding]).enter()\n\n// Removed the reference to the selectedEncoding variable\n .data(d => d.data).enter()\n\n~~~\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":219,"value":"md`Unfortunately, it turns out the Aaron relied on the data structure (and the use of a \\`selected_encoding\\` in a few places):\n- Defining scales for positioning the bars (\\`x\\`, \\`z\\`, and \\`clusterScale\\`)`","pinned":false,"hidden":false,"mode":"ojs"},{"id":224,"value":"md`So, we can define our own scales here:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":228,"value":"// Store a list of dates for our x scale\ndates = {\n // (need to find the min / max in the data)\n const all_dates = [];\n formatted_data.forEach(d => {\n d.data.forEach(dd => {\n if (all_dates.indexOf(dd.key) == -1) all_dates.push(dd.key);\n });\n });\n return all_dates.sort((a, b) => new Date(a) - new Date(b)); // sort!\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":239,"value":"// Get the maximum value (for the Y scale)\nmax_value = d3.max(formatted_data, d => d3.max(d.data.map(dd => +dd.value)))","pinned":true,"hidden":false,"mode":"ojs"},{"id":227,"value":"// Define our own x scale\nmy_x = d3\n .scaleBand()\n .domain(dates)\n .range([0, state_size])","pinned":true,"hidden":false,"mode":"ojs"},{"id":238,"value":"// Define our own y scale\nmy_y = d3\n .scaleLinear()\n .domain([0, max_value])\n .range([0, state_size + 100])","pinned":true,"hidden":false,"mode":"ojs"},{"id":204,"value":"// Importing my own version of the chart, with the data, labels, and scales specified\nimport { chart as forked_chart, state_size } with {\n formatted_data as cartogram_data,\n my_title as title,\n my_first_x_label as first_x_label,\n my_total_label as total_label,\n my_y_height_label as y_height_label,\n my_x as x,\n my_format as format,\n my_y as y\n} from \"@mkfreeman/small-multiple-chart-cartogram/2\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":210,"value":"html`Pro tip: If you are importing from another notebook, make sure to Publish it to see the changes reflected in your work`","pinned":false,"hidden":false,"mode":"ojs"},{"id":257,"value":"md`Still no luck! Turning out to be a bit tougher than I had hoped to put this data in that chart. To debug, I tried systematically commenting out sections of code in the \\`chart\\` method to find the following: \n\n**I was specifying the wrong variable name**`","pinned":false,"hidden":false,"mode":"ojs"},{"id":268,"value":"md`Oof -- this was a silly hole to fall into: I went looking for a variable called \\`data\\`, but the variable name was \\`cartogram_data\\`!!`","pinned":false,"hidden":false,"mode":"ojs"},{"id":272,"value":"md`With some unexpected time debugging, it took me about an hour and a half to get to this point of recreating the map (about **2 hours** up to this point, including writing it up):`","pinned":false,"hidden":false,"mode":"ojs"},{"id":471,"value":"displayImage(basic_cartogram)","pinned":false,"hidden":false,"mode":"ojs"},{"id":310,"value":"md`## The Details...`","pinned":false,"hidden":false,"mode":"ojs"},{"id":314,"value":"md`There are a number of really nice details about the NYT piece that allow it to express more data. These include:\n- Allowing the **New York state** data to extend upwards beyond it's state line (really drives home the high values)\n- Showing a yellow background on each state rectangle to show the **date of first case**\n- Displaying the date of first case below each state\n- Showing the **total cases** for each state\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":317,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":508,"value":"// Also setting the formatter\nmy_format = d3.format(\",\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":550,"value":"md`With that in place, I was able to get up to this point (now we're getting somewhere!)`","pinned":false,"hidden":false,"mode":"ojs"},{"id":553,"value":"displayImage(adjusted_scale)","pinned":false,"hidden":false,"mode":"ojs"},{"id":335,"value":"md`### First case background rectangles (~15 minutes)\nThe tricky part here was be determining when each state's first case is. This seems more appropriate to do here in the data than in the chart script. See the [formatting_data](#formatted_data) cell above for the implementation. Then, I was able to append rectangles in the script.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":557,"value":"displayImage(with_background_rects)","pinned":false,"hidden":false,"mode":"ojs"},{"id":350,"value":"md`### Add total cases (~10 minutes)\nEach state should also display a red piece of text with the total cases. This is also calculated in the the [formatting_data](#formatted_data) cell above.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":565,"value":"displayImage(with_text_labels)","pinned":false,"hidden":false,"mode":"ojs"},{"id":356,"value":"md`### Add date of first case for each state (~30 minutes)\nWell, we've got the starting point for each rectangle in the right place, now we need to add the date immediately below it. This means:\n- Adding a bit of spacing between the states (really just some futzing with the padding and height)\n- Appending and positioning text properly\n- Appending a tiny trianle for reading more precisely\n- Formatting the date to make it more human readable (maybe....)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":384,"value":"md`### Adding a legend (~45 minutes)\nReally, this is just creating an example state, and populating it with fake data (and adding some annotations). While it's tempting to do this by passing a new data object, I'm just going to create it manually on in the rendering function (on the first round). This required a bunch of copying and pasting, making it somewhat unweildy (hence, the refactoring described below).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":500,"value":"my_first_x_label = \"Date of first case\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":534,"value":"my_y_height_label = \"New cases each day\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":503,"value":"my_total_label = \"Total cases\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":394,"value":"md`### Adding a title (~10 minutes)\nThis really just required adding a \\`text\\` element and positioning it properly.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":491,"value":"my_title = \"Number of New Confirmed Cases Each Day\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":481,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":29,"value":"md`## Appendix`","pinned":false,"hidden":false,"mode":"ojs"},{"id":287,"value":"md`### Imports`","pinned":false,"hidden":false,"mode":"ojs"},{"id":33,"value":"import { displayImage } from \"@info474/utilities\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":97,"value":"import { statesData } from \"@mkfreeman/small-multiple-chart-cartogram/2\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":53,"value":"d3 = require(\"d3@v5\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"md`### Images`","pinned":false,"hidden":false,"mode":"ojs"},{"id":28,"value":"hopkins_map = await FileAttachment(\"hopkins_map.png\").url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":193,"value":"data_structure_labeled = await FileAttachment(\n \"data_structure_labeled.png\"\n).url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":124,"value":"data_structure = await FileAttachment(\"data_structure.png\").url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":468,"value":"adjusted_scale = await FileAttachment(\"adjusted_scale.png\").url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":546,"value":"basic_cartogram = await FileAttachment(\"basic_cartogram.png\").url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":559,"value":"with_background_rects = await FileAttachment(\"with_background_rects.png\").url()","pinned":false,"hidden":false,"mode":"ojs"},{"id":566,"value":"with_text_labels = await FileAttachment(\"with_text_labels.png\").url()","pinned":true,"hidden":false,"mode":"ojs"},{"id":302,"value":"md`#### Helpers, Styles`","pinned":false,"hidden":false,"mode":"ojs"},{"id":378,"value":"parseDate = d3.timeFormat(\"%b. %d\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":69,"value":"// Function to render a table\nrender_table = options => html`\n ${Object.keys(options[0]).map((d, i) => {\n let label = d;\n return ``;\n })}\n\n \n ${options.map(row => {\n return html`${Object.values(row).map(col => {\n return ``;\n })}`;\n })}\n \n

${label}
${col}

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"html``","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}