31:Tbfe,homework_summary = md` ----- ### Homework due 9/16 Just like last week, please log your time working on these assignments. If you find yourself spending more than 8 hours total on this work this week, **stop** and let Zach know. Note the time estimates provided below. ##### Logistics - Make sure you add your screenshots and links to the Week 3 Miro board. Take a look through other people's code! ##### Technical Learning (~5 hours) - Work through this whole page [tutorial](#tutorial_start), and complete the [2-part coding challenge](#challenge) described at the end. There's a lot this week, and we will review much of it in our classtime next week. Please go through it all and log any questions before our review discussion next week. It will take a few weeks for this all to stick, especially the data manipulation stuff, so please allow yourselves to be confused! Much of the process of learning to code is copying and pasting opaque code blocks, trying to make use of them, and watching everything break over and over again... ##### Exposure (~2 hour) - [DataStori.es Podcast](https://datastori.es/134-visualizing-uncertainty-with-jessica-hullman-and-matthew-kay/) interview with [Jessica Hullman](http://users.eecs.northwestern.edu/~jhullman/) and [Matthew Kay](http://www.mjskay.com) on visualizing uncertain data. - As introduced in class last week, please visit the many different election forecasting models and data visualization web sites linked on our Week 3 Miro board. All these sites are basically working to visualize and simulate the same exact political event, though each uses different methods and datasets to make their case. - Choose any two election visualizations, research their **methodologies and sources**, and create a short ~5 slide PDF comparing their overlap and differences. Consider how they talk about the upcoming election and their specific data predictions, the language on the pages, their persuasiveness and confidence, the scope and scale of their work, the visualization aesthetics itself, what kinds of visualization structures are used, what data they choose to show and where, how visible versus opaque their methods and sources are... Ultimately, which ones should we trust and why should we trust them, in their attempt to predict such an important world event? We will share these in small groups next week. ##### Ongoing Work (~30 minutes) - Continue adding unfamiliar terms to our class *glossary*, as well as logging to our list of *open questions* as you research and practice this week. Please, everyone, make an effort to add some questions and unusual terms or definitions! - Keep track in a bookmark folder any interesting data visualizations or interactive data experiences that you encounter. When we have any free time, we can share cool examples with one another! - Join Zach on Friday afternoon (by appointment, send an email to reserve a time), Monday morning (open access), or reach out for another time if you have any questions or would like to chat about data, design, dinosaurs... ----- `32:T931,d3_intro = md` ### Why D3? Having spent significant time covering the basics of the [SVG markup language (cheatsheet)](https://developer.mozilla.org/en-US/docs/Web/SVG/Element) and the [basic control structures and data flow](https://www.w3schools.com/js/js_loop_for.asp) facilitated by Javascript, we can turn our attention *finally* to D3 itself. We have talked up D3 alot. For more technical info, check out this [nice PDF](http://www.jeromecukier.net/presentations/d3-tutorial/S01%20-%20introduction.pdf) or this [Reddit AMA with Mike Bostock](https://www.reddit.com/r/dataisbeautiful/comments/3k3if4/hi_im_mike_bostock_creator_of_d3js_and_a_former/), the author of the libary. D3 is by far the most in-demand technical competency for data visualizers, and it is by a wide-margin the most commonly-used data visualization coding toolkit. The power and flexiblity of D3 is derived from how it *abstracts* away some of the fundamental structures of Javascript. We will no longer see \`\`for\`\` loops to iterate through datasets and manipulate arrays, but rather we will place our trust in D3 to contort the raw data behind the scenes. Our code will shrink in complexity and length, but significantly increase — dramatically on occasion — in *what the heck is going on in my own code?*. Debugging gets harder, as with abstraction comes necessary opacity. But, it's worth it — D3 automates so much of the slog of coding visualizations, and for many data visualizers, working without it would be akin to a communication designer losing InDesign. *But*, we still need to know how to do \`\`for\`\` loops and all of the other introductory material covered last week. D3 gives us a wide variety of tools to transform, mine, and smoothe out data — but it still requires native Javascript structures to guide its behaviors. For example, D3 can help us take a dataset of daily exercise duration and heart rate data, scale everything to pixels, choose *semantic* colors, and draw an interactive, animated, beautiful, and legible data visualization dashboard; but, it cannot (on its own) take that same dataset and **transpose** it into a list of heavy, medium, and light workouts. These *data synthesizing transformations* are often our goal as *design-led* data visualizers, and so we will need to return often to the swamps of native Javascript. `33:T8a6,dataset_intro = md` ### Dataset The exercises below use a dataset compiled by the [Global Biodiversity Information Facility](https://www.gbif.org), an international consortium of researchers, scientists, activists, and non-for-profit organizations that is [attempting to log all of the living species on our planet](https://www.gbif.org/what-is-gbif) while we undergo a [shocking mass-extinction event](https://livingplanet.panda.org/en-us/?utm_campaign=living-planet&utm_medium=media&utm_source=report). The dataset itself is a record of human- and machine-sightings of wildlife, logged with timestamps and geolocations, as well as other contextual information when relevant such as temperatures and elevation. The GBIF dataset is robust across species — ants to bears and squid to eagles — and also across time. GBIF members have worked through old newspapers and literature and tried to log every animal sighting for all of recorded human history. With that ambition. it is necessarily incomplete and inconsistent in places, but it is still an incredible resource for data scientists and data visualizers attempting to communicate the realities and threats facing the biodiversity of planet Earth. Please make an account on GBIF and explore the available data, it's truly incredible! Enter a species to search by on [this page](https://www.gbif.org/species/search), though you may need to look up the scientific name to find an animal you are interested in. Choose a favorite animal or family, and see what kinds of data might be available. When you find a species of interest, select it, and then click the "occurrences" button. You will then be able to filter your data like in the screenshot below. When you finish filtering, click "Download", and then choose "Simple." Depending on your selections, GBIF may then take a minute or two to prepare your data, and you will receive an email with a download link when it is ready to download as a CSV file. Specifically for this tutorial, we will be visualizing data based on the sightings of *monarch butterflies*, as observed *by humans*, from calendar years *2015 to 2020*, and only including observations logged *with geolocational coordinates*. ` 34:T471,md` \`\`FileAttachment(...)\`\` is specific to Observable — in other contexts we would see a complete URL there (\`\`"http://www......com"\`\`). \`\`await\`\` is a way to control the [**asynchronous**](https://nikgrozev.com/2017/10/01/async-await/) nature of contemporary Javascript. Back in the bad old days of web browsers being attached to slow computer processors and inconsistent dial-up network connections, it was decided that web browsers should run Javascript *as quick as possible*. So, if we tell a web browser to go get a file, it will go do that *and then continue running your code before the download completes*. For this reason, we cannot trust that a file is fully loaded unless we explicitly tell Javascript to wait for the download to complete. This is an annoying "feature," of Javascript, and will often give us headaches when dealing with external APIs and datasets. After *awaiting* the download, we *parse* the CSV file, casted as *text* (rather than the whole file as a complete blob of binary data). Now we have \`\`monarchs\`\` as a variable to look at. Pop it open by clicking its disclosure triangle. `35:T427,{ //svg variables let width = 720; let height = 360; //dataset cut let dataLimit = 2500; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //loop through all items in dataset for (let i = 0; i < dataLimit; i=i+1){ //cast to number, shift by 180 to remove negative numbers, double for desired size let xPosition = (parseFloat(monarchs[i].decimalLongitude) + 180) * 2 //cast to number, shift by 90 to remove negative numbers, double for desired size //subtract from height to flip upside down -- big latitudes become small, small become big let yPosition = height - (parseFloat(monarchs[i].decimalLatitude) + 90) * 2 //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", yPosition) .attr('r', 1) .attr('fill', 'orange') .attr('stroke', 'black') .attr('stroke-width', '.5') } //show visualization in Observable return svg.node(); }36:T6f2,md` Each of those dots is a butterfly sighting. Styled in orange and black, too cute? Note the clumping, as expected, in the eastern and central United States, as well as a nice trace of Australia's heavily-populated eastern coast, New Zealand, and Europe's southern bounds... and maybe one sighting in Hawaii? But... with a dataset that is supposed to be about *where butterflies are*, how is it that we can so easily make out the contours of North and Central America? Well, since this is a dataset of *human sightings of butterflies*, we would expect this dataset to be biased by where *humans* are! And so — as is often the case in attempting to visualize a map of some data phenomenon in the world — we accidentally make a map of human population density. Oops! Worse still, look at how filled-in California is! We can almost draw its border with Nevada and Oregon, though the border with Arizona is a bit more porous — are butterflies afraid of leaving the [Golden State](https://en.wikipedia.org/wiki/Golden_State)? Well, in an environmentally conscientious state like California, we might expect there to be more people participating in [citizen scientist](https://en.wikipedia.org/wiki/Citizen_science) initiatives like GBIF. So, not only are human *locations* distorting this data, but also human *preferences, proclivities, and beliefs* are throwing us off. We will cover some statistical weighting methods to offset this bias in the future, but for now, let's move on and acknowledge that butterflies are rarely sea-faring (setting aside the totally unrelated [sea butterfly](https://en.wikipedia.org/wiki/Sea_butterfly)), and so their migrations would likely be [coterminous](https://en.wiktionary.org/wiki/coterminous) with heavily-populated landmasses. `37:T97b,future_improvements = md` ### Improvements There is some obvious awkwardness in the code and visualization above. - The mathematical functions that locate the long-lat circles on our SVG artboards are... inelegant. Also, no shame, we do not want to do that kind of math ever again. - We are assigning the dots a universal color. It would be nice if we could color by month or state, for example. In other words, we have no way to treat specific datapoints within the dataset uniquely with respect to any SVG attribute or style. - The culling of the dataset is arbitrary. A more appropriate trim of a medium-large dataset like this might be to look at just a month at a time, or just one country/state/province. Still, the huge number of SVG circles within North America is a problem, as we cannot visually gauge how many dots show up due to the layering. A naive approach would be to add some transparency to the dots, but that doesn't really help at this scale and just muddies the map (try it above though, perhaps you'll have better luck with styling). - If we wanted to help save monarch butterflies, we might need to persuade certain civic bodies. Accordingly, our map of the natural world might need to be — for persuasion's sake — distorted by human social constructions. We might, for example, make a [choropleth map](https://en.wikipedia.org/wiki/Choropleth_map) of the counties of the United States, colored by their butterfly counts, in order to engage county-level environmental conservation bodies. In other words, the long-lat coordinates of these butterflies might not be as useful to communicate — and we might instead want to count up the butterfly sightings that fall into differently-shaped buckets like geographic counties, seasons, months. Yes, this is a description of scary [histograms](https://en.wikipedia.org/wiki/Histogram). This would also let us compare this dataset against others, like a dataset on average temperatures or human density, more easily. - Once we have drawn the dot, we do not know what data it relates to. If we wanted a [tooltip](https://en.wikipedia.org/wiki/Tooltip) to show up when the user clicks on a dot and display the date of observation, we have no way of working backwards from the visualized SVG \`\`\`\` to a row of data. We will improve our code to address each of these issues in turn, except the last one, which will be left for next week. `38:T4f2,scales = md` ### D3 Scales Converting data units to visualization units consistently and controllably is at the core of data visualization. [D3 scales](https://github.com/d3/d3-scale) let us do that. Scales are functions that have an input **domain** and an output **range**, and all they do is convert numbers from one to the other. The example scale below takes numbers between 0 and 10, and returns numbers between 200 and 600. 0 becomes 200, 10 becomes 200, 5 becomes 400, and everything else *scales* proportionally. \`\` let exampleScale = d3.scaleLinear().domain([0,10]).range([200,600]); \`\` The domain is the low and the high number that define the boundaries of the input data to be considered. The range, similarly, sets the low and the high number that define the boundaries of the data spit out. Most often, the domain is defined by a dataset's units, and the range is set to the pixels available in visualization space. \`\` let convert5 = exampleScale(5) // convert5 would equal 400 \`\` Note that scales (by default) do not **clamp**. We can ask the scale above to transform -3, which is smaller than the lowest number in our domain, and the scale will return 80, which is *proportionally* smaller than the smallest number in our range. `39:T580,{ //svg variables let width = 800; let height = 100; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //simple data let exampleData = [9,2,7.1,5,0,1,6,3.15,-3,13]; //create scale to convert data to pixels let exampleScale = d3.scaleLinear().domain([0,10]).range([200,600]); //draw horizontal line svg.append("line") .attr("x1", 200) .attr("y1", height/2) .attr("x2", 600) .attr("y2", height/2) .attr('fill', 'none') .attr('stroke', 'black') .attr('stroke-width', '2') ; //loop through all items in dataset for (let i = 0; i < exampleData.length; i=i+1){ //convert data to pixels let xPosition = exampleScale(exampleData[i]) //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height/2) .attr('r', 5) .attr('fill', 'orange') .attr('stroke', 'black') .attr('stroke-width', '2') //draw text for each item svg.append("text") .attr("x", xPosition) .attr("y", (height/2) - 20 ) .style('font-family','courier') .style('font-size',10) .style('text-anchor','middle') .text( exampleData[i] + "→" + exampleScale(exampleData[i])) } //show visualization in Observable return svg.node(); }3a:T586,{ //svg variables let width = 800; let height = 100; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //dataset let exampleData = [1,10,100,1000,10000,100000] //create scale to convert data to pixels let exampleScale = d3.scaleLog().domain([1,10000]).range([200,600]) //draw horizontal line svg.append("line") .attr("x1", 200) .attr("y1", height/2) .attr("x2", 600) .attr("y2", height/2) .attr('fill', 'none') .attr('stroke', 'black') .attr('stroke-width', '2') ; //loop through all items in dataset for (let i = 0; i < exampleData.length; i=i+1){ //convert data to pixels let xPosition = exampleScale(exampleData[i]) //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height/2) .attr('r', 5) .attr('fill', 'orange') .attr('stroke', 'black') .attr('stroke-width', '2') //draw text for each item svg.append("text") .attr("x", xPosition) .attr("y", (height/2) - 20 ) .style('font-family','courier') .style('font-size',10) .style('text-anchor','middle') .text( exampleData[i] + "→" + Math.round(exampleScale(exampleData[i]))) } //show visualization in Observable return svg.node(); }3b:T582,{ //svg variables let width = 800; let height = 100; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //dataset let exampleData = [0,1,2,3,4,5,6,7,8,9,10] //create scale to convert data to pixels let exampleScale = d3.scaleLinear().domain([0,10]).range([600,200]) //draw horizontal line svg.append("line") .attr("x1", 200) .attr("y1", height/2) .attr("x2", 600) .attr("y2", height/2) .attr('fill', 'none') .attr('stroke', 'black') .attr('stroke-width', '2') ; //loop through all items in dataset for (let i = 0; i < exampleData.length; i=i+1){ //convert data to pixels let xPosition = exampleScale(exampleData[i]) //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height/2) .attr('r', 5) .attr('fill', 'orange') .attr('stroke', 'black') .attr('stroke-width', '2') //draw text for each item svg.append("text") .attr("x", xPosition) .attr("y", (height/2) - 20 ) .style('font-family','courier') .style('font-size',10) .style('text-anchor','middle') .text( exampleData[i] + "→" + Math.round(exampleScale(exampleData[i]))) } //show visualization in Observable return svg.node(); }3c:T655,{ //svg variables let width = 800; let height = 100; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //create empty dataset let exampleData = [] //fill up random dataset for (let i = 0; i < 5; i = i+1){ exampleData.push(Math.random()*50) } //extract the min and the max of the dataset let exampleExtent = d3.extent(exampleData) //create scale to convert data to pixels let exampleScale = d3.scaleLinear().domain(exampleExtent).range([200,600]) //draw horizontal line svg.append("line") .attr("x1", 200) .attr("y1", height/2) .attr("x2", 600) .attr("y2", height/2) .attr('fill', 'none') .attr('stroke', 'black') .attr('stroke-width', '2') ; //loop through all items in dataset for (let i = 0; i < exampleData.length; i=i+1){ //convert data to pixels let xPosition = exampleScale(exampleData[i]) //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height/2) .attr('r', 5) .attr('fill', 'orange') .attr('stroke', 'black') .attr('stroke-width', '2') //draw text for each item svg.append("text") .attr("x", xPosition) .attr("y", (height/2) - 20 ) .style('font-family','courier') .style('font-size',10) .style('text-anchor','middle') .text( exampleData[i].toFixed(1) + "→" + exampleScale(exampleData[i]).toFixed(1)) } //show visualization in Observable return svg.node(); }3d:T686,color_scales = md ` ### Color Scales Another kind of scale, which we ❤️ above most others, is the ability for D3 to take a number and spit out a color proportionally, in the same way we spit out pixel values proporationally with regular scales. This lets us use color *semantically* - color with meaning! These are called *color scales*, and there are a bunch of [built-in gradients](https://observablehq.com/@d3/color-schemes) (important link!) that we can sample, as well as the ability to make our own. Normally, we use the built-in color scales since they were [designed with data](https://colorbrewer2.org/#type=sequential&scheme=BuGn&n=3) to be *perceptually-uniform*. We will do a deep dive into *semantic* color choice in future weeks, as it is a complicated and important topic worthy of coversation and additional study. The names of these scales are universal across many different libraries and programming languages, and are in many ways a shared language across all data visualization practitioners. And, everyone loves *viridis*. Color scales are used just like linear or exponential scales. But, we cannot set the domain or range of a color scale — so, we need to instead contort our data to fit with what color scales expect, which is a number between 0 and 1. We call such values, defined as a position between 0 and 1, a **parameter**. So, we often need to make a scale for the color scales. Sigh. Change the value fed to the scale below to generate data-driven color values. Note that color scales clamp! Any number below or above the [0,1] domain of the color scale maps to the lowest or highest value in the gradient, respectively. ` 3e:T8ee,{ //svg variables let width = 800; let height = 100; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) //simple data let exampleData = [1,2,3,4,5,6,7,8,9]; //create scale to convert data to pixels let pixelScale = d3.scaleLinear().domain([0,10]).range([200,600]); //create scale to convert data to number between 0 and 1 let parameterScale = d3.scaleLinear().domain([0,10]).range([0,1]); let scaleNames = ["viridis","magma","cividis"]; for(let i = .25; i <= .75; i=i+.25){ //draw horizontal line svg.append("line") .attr("x1", 200) .attr("y1", height*i) .attr("x2", 600) .attr("y2", height*i) .attr('fill', 'none') .attr('stroke', 'black') .attr('stroke-width', '2') ; //draw label svg.append("text") .attr("x", 190) .attr("y", height*i) .style('font-family','courier') .style('font-size',10) .style('text-anchor','end') .style('alignment-baseline','middle') .text(scaleNames[ (i*4) - 1]) ; } //loop through all items in dataset for (let i = 0; i < exampleData.length; i=i+1){ //convert data to pixels let xPosition = pixelScale(exampleData[i]) //convert data to number between 0 and 1 let parameterData = parameterScale(exampleData[i]) //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height*.25) .attr('r', 10) .attr('fill', d3.interpolateViridis(parameterData)) .attr('stroke', 'black') .attr('stroke-width', '2') //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height*.5) .attr('r', 10) .attr('fill', d3.interpolateMagma(parameterData)) .attr('stroke', 'black') .attr('stroke-width', '2') //draw circle for each item svg.append("circle") .attr("cx", xPosition) .attr("cy", height*.75) .attr('r', 10) .attr('fill', d3.interpolateCividis(parameterData)) .attr('stroke', 'black') .attr('stroke-width', '2') } //show visualization in Observable return svg.node(); }3f:T54b,{ //svg variables let width = 720; let height = 360; //dataset cut let dataLimit = 5000; //create SVG artboard let svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); let bg = svg .append('rect') .attr("x", 0) .attr("y", 0) .attr("width", width) .attr("height", height); //scales for converting geographic coordinates to pixels let longitudeScale = d3 .scaleLinear() .domain([-150, -60]) .range([0, width]); let latitudeScale = d3 .scaleLinear() .domain([10, 55]) .range([height, 0]); //scale for converting months to a number between 0 and 1 let monthScale = d3 .scaleLinear() .domain([1, 12]) .range([0, 1]); //loop through all items in dataset for (let i = 0; i < dataLimit; i = i + 1) { let xPosition = longitudeScale(parseFloat(monarchs[i].decimalLongitude)); let yPosition = latitudeScale(parseFloat(monarchs[i].decimalLatitude)); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", yPosition) .attr('r', 1.4) .attr( 'fill', d3.interpolatePlasma(monthScale(parseInt(monarchs[i].month))) ) .attr('opacity', .7); } //show visualization in Observable return svg.node(); }40:T53d,conditionals = md` ### Conditionals So, let us imagine a very simple goal. What if we wanted to be able to show the number of butterfly sightings *per year* as a simple bar chart? ##### Simple Conditions In the code below, note the basic [**conditional** control structure](https://www.javascript.com/learn/conditionals) that has not been covered formally yet \`\`if(){}, else if(){}, and else\`\`. Their behaviors are straightforward — each \`\`if\`\` checks whether or not a simple question is true or not. Less than (<), greater than (>), less than or equal to (<=), greater than or equal to (>=), equal to (==), and not equal to (!=) are all options. Note the *double* equal sign (==) on equals to, it is easy to forget. In the event of the condition being true, the code within the braces is run. Otherwise, that code is skipped. We can use this simple branching logic to build a new dataset from our existing butterfly data. The code below uses the shorthand \`\`++\`\`, called the **increment** operator but more often pronounced "plus plus". \`\`variable++\`\` is just a shorter and more idiomatic way of writing \`\`variable = variable + 1\`\`, and simply adds one to the variable that it follows. **Decrement** is written \`\`--\`\` and pronounced "minus minus" and works the same way, but subtracts one rather than adds. `41:Tc40,{ //collector counting off the months let months = {} //look over for(let i = 0; i05 but 11->11 if (parseInt(monarchs[i].month) < 10){ sightingMonth = monarchs[i].year + "0" + monarchs[i].month; } else{ sightingMonth = monarchs[i].year + monarchs[i].month; } //check if we have seen this month+year combo before if (sightingMonth in months){ //if we have seen it already, add one months[sightingMonth]++ } else { //otherwise, make a new key months[sightingMonth] = 1; } } //create an arrays to organize data let butterfliesByMonth = []; for (let property in months) { butterfliesByMonth.push( {name:property, count:months[property]} ) } //sort months by their name - weird and not covered in the tutorial, but should make some intuitive sense? //looks like an accessor function, right? butterfliesByMonth.sort((a, b) => { return parseInt(a.name) - parseInt(b.name); }); //svg variables let width = 800; let height = 400; let margin = 25; //create SVG artboard let svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width",width) .attr("height",height) .attr("fill","#ddd") //create SVG background let bg = svg.append("rect") .attr('x',0) .attr('y',0) .attr("width",width) .attr("height",height) //accessor function to find min and max counts let monthMinMax = d3.extent(butterfliesByMonth, d => d.count); //scales for counts to pixels and parameters let monthSizeScale = d3.scaleLinear().domain(monthMinMax).range([margin,height-(margin*2)]); let monthParameterScale = d3.scaleLinear().domain(monthMinMax).range([0,1]); //how wide should each bar be? let barWidth = (width - (margin * 2)) / butterfliesByMonth.length; //loop through all items in dataset for (let i = 0; i < butterfliesByMonth.length; i++){ // for converting data to pixels let barLength = monthSizeScale(parseFloat(butterfliesByMonth[i].count)) // for converting data to parameter between 0 and 1 let barParameter = monthParameterScale(parseFloat(butterfliesByMonth[i].count)) //draw rectangle for each month svg.append("rect") .attr("x", (i*barWidth) + margin) //pesky needing to flip vertically .attr("y", (height - barLength) - margin) .attr('width', barWidth) .attr('height', barLength) .attr('fill', d3.interpolatePlasma( barParameter)) //check if we are seeing a January... if ( butterfliesByMonth[i]["name"].slice(4) == "01"){ //add a textlabel for the new year svg.append("text") .attr("x", (i*barWidth) + margin) .attr("y", height - (margin/2)) .text( butterfliesByMonth[i]["name"].slice(0,4) ) .attr('fill','black') .attr('font-family','courier') .attr('font-size',8) } } //show visualization in Observable return svg.node(); } 42:[{"id":"338e791ff714ff46d1672fc7c3660befd6acc0a5762ee3e0b5bb2600989e2383f8f29d98c82c5916e5b355bfe4c6f52b8447183d859a31be12440b093d5af1a1","href":"https://static.observableusercontent.com/files/338e791ff714ff46d1672fc7c3660befd6acc0a5762ee3e0b5bb2600989e2383f8f29d98c82c5916e5b355bfe4c6f52b8447183d859a31be12440b093d5af1a1","name":"gbif.png","type":"image/png","lastModified":1599749164306.945,"size":1086219},{"id":"da64d86c466399e4f509600d88e3a35bf83062ab485a8d9f606ba58ccbe94a9c1ca006fe1ad0979c7ad0aeef3d22afaf5a2ee072315c13efa9beba0b131c54ab","href":"https://static.observableusercontent.com/files/da64d86c466399e4f509600d88e3a35bf83062ab485a8d9f606ba58ccbe94a9c1ca006fe1ad0979c7ad0aeef3d22afaf5a2ee072315c13efa9beba0b131c54ab","name":"butterflies.csv","type":"text/csv","lastModified":1599749989177.737,"size":12612777}] 2f:{"title":"Week 3 : Intro to D3 and Data Synthesis","cells":[{"id":0,"value":"title = md`\n# Week 3 : Intro to D3 and Data Synthesis\n## Transposing Datasets, Scaling Data, and D3 Charting\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":936,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"FileAttachment(\"gbif.png\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"md`\n65,000+ butterfly sightings, ready for us to visualize! \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":820,"value":"butterflies = md\n`\n### Why Butterflies?\n\nThey're [beautiful and weird](https://en.wikipedia.org/wiki/Monarch_butterfly ). They're [picky eaters of a poisonous plant](https://www.saveourmonarchs.org/why-milkweed.html). They [travel incredibly far on tiny wings](https://www.nationalgeographic.com/news/2017/10/monarch-butterfly-migration/). They're [at risk](https://www.worldwildlife.org/species/monarch-butterfly). They're Illinois' [official insect](https://statesymbolsusa.org/symbol-official-item/illinois/state-insect/monarch-butterfly). They're important enough for [Chicago to dedicate several acres of lakefront garden](https://www.fieldmuseum.org/science/research/area/keller-science-action-center/science-action-chicago/monarchs-view-city) for their preservation. They're [easy to see and also easy to forget](https://www.chicagotribune.com/news/sns-wp-washpost-bc-butterflies09-20150209-story.html). 🦋\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":47,"value":"dataset_analysis = md`\n### Dataset Analysis\n\nSo, what the heck is in this dataset? Take a look at the array of Javascript objects that has been minimally cleaned-up for purposes of simplicity and file size. \n\nBut, first, we absolutely must load D3.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":55,"value":"d3 = require(\"d3@5\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":61,"value":"md`\nAnd now, loading the dataset from an attached CSV (comma-separated values) file with D3. CSVs are a simple data interchange format, basically an Excel file with the columns separated by commas and the rows entered on their own lines.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":63,"value":"monarchs = d3.csvParse(await FileAttachment(\"butterflies.csv\").text())","pinned":true,"hidden":false,"mode":"ojs"},{"id":66,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"monarchs","pinned":true,"hidden":false,"mode":"ojs"},{"id":85,"value":"md`\nTons of data! ~65,000 entries of butterfly sightings with unique identifiers, dates, coordinates, country and state codes, and sighting metadata such as the individual or organization that logged this butterfly and their identifying code within GBIF.\n\nYou can load this array into your own Observable notebook with the following line placed in a code block.\n\n\\`\\`\nimport {monarchs} from '@zachpino/week-3-intro-to-d3-and-data-synthesis'\n\\`\\`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":98,"value":"simple_map = md`\n### Simple Map\n\nLet's draw a map like we did last week. To prevent this code from crashing our web browsers, we will only look at the first 2,500 butterfly sightings. We will need to take special precautions in the future to manage this much data, but for learning purposes, this simple data trim is forgivable — especially on this tutorial page which needs to manage multiple maps.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":104,"value":"$35","pinned":true,"hidden":false,"mode":"ojs"},{"id":112,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":121,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":200,"value":"$39","pinned":true,"hidden":false,"mode":"ojs"},{"id":187,"value":"exotic_scales = md`\nMost often, we use linear scaling as above, which transforms numbers in one domain to a different range uniformly. Also available to us, amongst others, are logarithmic and exponential scales, which have been used throughout 2020 to communicate viral spread to middling effect. [Humans are bad at intuiting exponential growth.](https://www.pnas.org/content/117/28/16264)\n\nWe can think about these other more exotic scales as scaling each number differently, depending on its magnitude. Here, even though the numbers are spread out by different amounts from one another, they show up evenly spaced on the line.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":254,"value":"$3a","pinned":true,"hidden":false,"mode":"ojs"},{"id":377,"value":"inverting_scales = md`\nThe order of values entered as domains and ranges is meaningful. This makes it possible to flip-around datasets, like we saw possible with that confusing subtraction exercise last week. Might be hard to see the difference in the code below, but the only change is in order of the items in \\`\\`.range()\\`\\`. Big and then small, rather than the usual small and then big.\n\n\\`\\`\nlet exampleScale = d3.scaleLinear().domain([0,10]).range([600,200])\n\\`\\`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":383,"value":"$3b","pinned":true,"hidden":false,"mode":"ojs"},{"id":298,"value":"statistical_tools = md`\n### Statistical Tools\nVery often, we do not know the bounding low and high numbers of a dataset — especially when ingesting dynamic or live data. So, it can be helpful to extract those values programmatically. This is a very common coding pattern when working with scales, and so D3 provides [excellent statistical utilities](https://observablehq.com/@d3/d3-extent) for extracting meaningful data from a dataset. Take a look at the code below, click through its output, and run it a few times to see the numbers change.\n\nTake note specifically of \\`\\`d3.extent()\\`\\`. It gives us everything we need to setup scale domains in exactly the right format (a two item array)!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":302,"value":"{\n //create empty dataset\n let exampleData = [];\n \n //fill up random dataset\n for (let i = 0; i < 10; i = i+1){\n exampleData.push(Math.random()*50);\n }\n\n let min = d3.min(exampleData);\n let max = d3.max(exampleData);\n let mean = d3.mean(exampleData);\n let extent = d3.extent(exampleData);\n\n return {minimum: min, maximum: max, average: mean, minmax: extent, data: exampleData}\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":320,"value":"md`\nAnd, we can combine extents and scales like this. Note that the lowest number of the dataset will transform into the lowest number of the range, so this approach automatically **clamps** the data. Data will never fall outside of the range, which is often the goal (but not always!).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":277,"value":"$3c","pinned":true,"hidden":false,"mode":"ojs"},{"id":896,"value":"accessor_functions = md`\nLet's make a small addition to the above, one which will come back to haunt us again and again. What if we wanted to find the westernmost, easternmost, southernmost, and northernmost monarch sightings? \n\nWe cannot just ask for the \\`\\`d3.max()\\`\\` of the \\`\\`monarchs\\`\\` array, as every row has a ton of different data. How would JS know what to compare? Instead, we use **accessor functions**, which are the D3 method for burrowing into a dataset, without us needing to write a \\`\\`for\\`\\` loop.\n\nWe will come back to this syntax soon, but for now read it as \"look at the \\`\\`monarchs\\`\\` array, and taking each one as \\`\\`d\\`\\`, find their minimum and maximum longitudes/latitudes...\"\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":903,"value":"{\n return d3.extent(monarchs, d => d.decimalLongitude)\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":910,"value":"{\n return d3.extent(monarchs, d => d.decimalLatitude)\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":327,"value":"scaled_map = md`\n### Back to Butterflies\nLet's use scales to clean up our map code. We will not use \\`\\`d3.extents()\\`\\` here since we know the domain of longtiude and latitude lines, [-180 to 180] and [-90 to 90]. \n\nNevertheless, much cleaner code! No more opaque arithmetic to manage.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":197,"value":"{\n //svg variables\n let width = 720;\n let height = 360;\n \n //dataset cut\n let dataLimit = 5000;\n \n //create SVG artboard\n let svg = d3.create(\"svg\")\n .attr(\"viewBox\", [0, 0, width, height])\n .attr(\"width\",width)\n .attr(\"height\",height)\n\n //scales for converting geographic coordinates to pixels\n let longitudeScale = d3.scaleLinear().domain([-180,180]).range([0,width]);\n let latitudeScale = d3.scaleLinear().domain([-90,90]).range([height,0]);\n \n //loop through all items in dataset\n for (let i = 0; i < dataLimit; i=i+1){\n \n let xPosition = longitudeScale(parseFloat(monarchs[i].decimalLongitude))\n let yPosition = latitudeScale(parseFloat(monarchs[i].decimalLatitude))\n\n //draw circle for each item\n svg.append(\"circle\")\n .attr(\"cx\", xPosition) \n .attr(\"cy\", yPosition)\n .attr('r', 1)\n .attr('fill', 'orange')\n .attr('stroke', 'black')\n .attr('stroke-width', '.5')\n }\n \n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":339,"value":"zoom_and_crop = md`\n### Zooming and Cropping\nThere are definitely better ways to do this, but we can use scales to do a naive zooming and cropping of the map. If we wanted only to look at North America, we could limit our input geographic domains to just [-180 to 0] for longitude and [0 to 90] for latitude. Note that we are still drawing all the other dots, they are just off-screen.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":341,"value":"{\n //svg variables\n let width = 720;\n let height = 360;\n \n //dataset cut\n let dataLimit = 5000;\n \n //create SVG artboard\n let svg = d3.create(\"svg\")\n .attr(\"viewBox\", [0, 0, width, height])\n .attr(\"width\",width)\n .attr(\"height\",height)\n\n //scales for converting geographic coordinates to pixels\n let longitudeScale = d3.scaleLinear().domain([-180,0]).range([0,width]);\n let latitudeScale = d3.scaleLinear().domain([0,90]).range([height,0]);\n \n //loop through all items in dataset\n for (let i = 0; i < dataLimit; i=i+1){\n \n let xPosition = longitudeScale(parseFloat(monarchs[i].decimalLongitude))\n let yPosition = latitudeScale(parseFloat(monarchs[i].decimalLatitude))\n\n //draw circle for each item\n svg.append(\"circle\")\n .attr(\"cx\", xPosition) \n .attr(\"cy\", yPosition)\n .attr('r', 1)\n .attr('fill', 'orange')\n .attr('stroke', 'black')\n .attr('stroke-width', '.5')\n }\n \n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":356,"value":"md`\nBe careful, though. Because we are using two scales, it is very easy to warp the map if we adjust the scales differently. Of course, like in all things data viz, sometimes this is [desirable](https://www.esri.com/news/arcuser/0110/cartograms.html) and unavoidable when taking the 3D sphere of the Earth and [flattening](https://en.wikipedia.org/wiki/Map_projection) it into a 2D map.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":354,"value":"{\n //svg variables\n let width = 720;\n let height = 360;\n \n //dataset cut\n let dataLimit = 5000;\n \n //create SVG artboard\n let svg = d3.create(\"svg\")\n .attr(\"viewBox\", [0, 0, width, height])\n .attr(\"width\",width)\n .attr(\"height\",height)\n\n //scales for converting geographic coordinates to pixels\n let longitudeScale = d3.scaleLinear().domain([-140,-60]).range([0,width]);\n let latitudeScale = d3.scaleLinear().domain([0,90]).range([height,0]);\n \n //loop through all items in dataset\n for (let i = 0; i < dataLimit; i=i+1){\n \n let xPosition = longitudeScale(parseFloat(monarchs[i].decimalLongitude))\n let yPosition = latitudeScale(parseFloat(monarchs[i].decimalLatitude))\n\n //draw circle for each item\n svg.append(\"circle\")\n .attr(\"cx\", xPosition) \n .attr(\"cy\", yPosition)\n .attr('r', 1)\n .attr('fill', 'orange')\n .attr('stroke', 'black')\n .attr('stroke-width', '.5')\n }\n \n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":401,"value":"$3d","pinned":false,"hidden":false,"mode":"ojs"},{"id":420,"value":"{ \n let parameterScale = d3.scaleLinear().domain([0,10]).range([0,1]);\n \n return d3.interpolateViridis(parameterScale(5));\n}\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":457,"value":"md`\nTo clarify, the data flow is like this...\n\n\\`\\`\\`\nnumbers between a [min,max] -> \n -> regular scale with domain [min,max] and range [0,1] -> \n -> scaled numbers between [0,1] -> \n -> choice of color scale \n -> color values\n\\`\\`\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":450,"value":"$3e","pinned":true,"hidden":false,"mode":"ojs"},{"id":522,"value":"map_with_colorscales = md`\n### Back to Butterflies Again!\nLet's use color scales to improve our map code again. We can use the month encoded in each butterfly observation to generate a color for each dot.\n\nNow we can see the effect of seasonal butterfly migration in the color of the dots! Zoom in close to see the color variation.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"$3f","pinned":true,"hidden":false,"mode":"ojs"},{"id":560,"value":"md`\nCool.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":601,"value":"transposing = md`\n### Beyond Maps \nSo, we made a prettier map than when we started. We cleaned-up our code with scales, zoomed in, and added some semantic color. And, to do anything else from our list of improvements, we need to in some way **transpose** our data. Some of the columns in our butterfly dataset need to become rows, and vice-versa.\n\nThere are many ways to do these sorts of techniques in Javascript, and often we need to do these operations on thousands of entries and so speed absolutely matters. The approaches below are chosen for ease-of-understanding and not speed, and we will likely replace them later as our competencies develop.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":578,"value":"$40","pinned":false,"hidden":false,"mode":"ojs"},{"id":586,"value":"{\n let year2015 = 0;\n let year2016 = 0; \n let year2017 = 0; \n let year2018 = 0; \n let year2019 = 0; \n let year2020 = 0; \n let yearOther = 0;\n \n for(let i = 0; i