31:T92c,d3_intro = md` ### Why D3? Having spent significant time covering the basics of the [SVG markup language (cheatsheet)](https://developer.mozilla.org/en-US/docs/Web/SVG/Element) and the [basic control structures and data flow](https://www.w3schools.com/js/js_loop_for.asp) facilitated by Javascript, we can turn our attention *finally* to D3 itself. We have talked up D3 alot. For more technical info, check out this [nice intro](https://homes.cs.washington.edu/~jheer/talks/VisWeek2012/#4) or this [Reddit AMA with Mike Bostock](https://www.reddit.com/r/dataisbeautiful/comments/3k3if4/hi_im_mike_bostock_creator_of_d3js_and_a_former/), the author of the libary. D3 is by far the most in-demand technical competency for data visualizers, and it is by a wide-margin the most commonly-used data visualization coding toolkit. The power and flexiblity of D3 is derived from how it *abstracts* away some of the fundamental structures of Javascript. We will no longer see \`\`for\`\` loops to iterate through datasets and manipulate arrays, but rather we will place our trust in D3 to contort the raw data behind the scenes. Our code will shrink in complexity and length, but significantly increase — dramatically on occasion — in *what the heck is going on in my own code?*. Debugging gets harder, as with abstraction comes necessary opacity. But, it's worth it — D3 automates so much of the slog of coding visualizations, and for many data visualizers, working without it would be akin to a communication designer losing InDesign. *But*, we still need to know how to do \`\`for\`\` loops and all of the other introductory material covered over the last few weeks. D3 gives us a wide variety of tools to transform, mine, and smoothe out data — but it still requires native Javascript structures to guide its behaviors. For example, D3 can help us take a dataset of daily exercise duration and heart rate data, scale everything to pixels, choose *semantic* colors, and draw an interactive, animated, beautiful, and legible data visualization dashboard; but, it cannot (on its own) take that same dataset and **transpose** it into a list of heavy, medium, and light workouts. These *data synthesizing transformations* are often our goal as *design-led* data visualizers, and so we will need to return often to the swamps of native Javascript. `32:T89b,md` ---- ### Querying Data from an API: US Census We have so far been been accustomed to work with data that can be encapsulated in a CSV file or typed into a Javascript array. Contemporary data work, however, often requires that we interact with live data stored on a remote server, which might be much larger than we could ever conveniently hold in a single file. When companies and organizations want to make such large datasets available, they often offer a public **API**, an *application programming interface*, to give the world access to that data. APIs are conceptually very simple: each API is a structured URL pattern, set of data description conventions, and access protocols for filtering, sorting, and manipulating or submitting data. In practice, though their are many competing attempts at [API standardization](https://swagger.io), almost every API you encounter will be a unique challenge to overcome. Many web services offer data (free or for a fee) via an API. Take a look at this [awesome list of free APIs](https://github.com/public-apis/public-apis) for us to reference in the future! We will spend much more time on the structure of these API queries in future weeks, but for now, let's just grab some data and get to some visualization coding ASAP. The code below \`awaits\` data from the US census, t in particular the [yearly American Community Survey (ACS) API](https://www.census.gov/data/developers/data-sets/acs-1year.html) from the year \`2024\` to give us data relating to [census variable](https://api.census.gov/data/2021/acs/acs1/variables.html) \`B01001_001E\` (total population) and the \`NAME\`, for all (\`*\`) the \`states\`. For more complex queries of the US Census API, we would need a census key. If you [request one](https://api.census.gov/data/key_signup.html) (yes, this weirdly basic page is a US federal website!), it is important that you not publish it on public websites to prevent abuse. We will spend more time on this API query structure in the future, but if you are curious, you can explore how to access data with more specificity by browsing the [examples](https://api.census.gov/data/2021/acs/acs1.html) provided by the US government. `33:T67c,md` Take a look at the resulting data that comes back from the US government... notice anything strange? For one, the data is clearly structured as a CSV file with a header row, and not as a Javascript object. That's annoying. The first item in the returned array is the header row, and each state after is an array. But we have 53 items, and not the 51 we would expect (50 states + 1 header row). This is because, for census purposes, Washington DC and Puerto Rico are considered states. Each state array has 3 data points. The \`name\` of the state, code \`B01001_001E\` which is the total population of the state (as a string!), and finally a \`state\` code, which we did not ask for, representing the [FIPS code](https://www.nrcs.usda.gov/wps/portal/nrcs/detail/?cid=nrcs143_013696) for the state. These codes are very useful for aligning data between different sources. A quick note on that \`await\` keyword, which allows for control over the [**asynchronous**](https://nikgrozev.com/2017/10/01/async-await/) nature of contemporary Javascript. Back in the bad old days of web browsers being attached to slow computer processors and inconsistent dial-up network connections, it was decided that web browsers should run Javascript *as quick as possible*. So, if we tell a web browser to go get a file, it will go do that *and then continue running your code before the download completes*. For this reason, we cannot trust that a file is fully loaded unless we explicitly tell Javascript to wait for the download to complete. This is an annoying "feature," of Javascript, and will often give us headaches when dealing with external APIs and datasets. `34:T809,improvements = md` ----- ### Improvements There is some obvious awkwardness in the code and visualizations above. - Depending on the current state of the census API and the particulars of your query, the sorting of the states may be weird. Sometimes you will see Mississippi first. Sometimes Alabama. Sometimes California. Often, you'll see DC listed last as "District of Columbia", and other times it and Puerto Rico will be alphabetically interleaved with the others. Parts of the sequence are sometimes alphabetical, and other parts might just be random shuffled. Because we are looping through the states in the order that they came back from the census, our visualization's sorting order is bound by the (absent) logic of the data provider! - The mathematical functions that locate and size the circles and rectangles in the SVG artboards are... inelegant. Also, no shame, we do not want to do that kind of math ever again. And, the math does not really work anyways since everything is colliding and hard to read! The [magic number](https://en.wikipedia.org/wiki/Magic_number_%28programming%29) \`\`100000\`\`, used to scale the circles to pixels, was arrived at experimentally and doesn't work very well — there must be a better way! - We are assigning the dots and rects a universal color. It would be nice if we could color by some other data. In other words, we have no way to treat specific datapoints within the dataset uniquely with respect to any SVG attribute or style. - The state name labels are ugly, poorly placed, and intersecting. Many are too long to be placed cleanly. - Once we have drawn the SVG shapes, we do not know what real data it relates to. If we wanted a [tooltip](https://en.wikipedia.org/wiki/Tooltip) to show up when the user clicks on a shape and display the actual number, we have no way of working backwards from the visualized SVG \`\`\`\` or \`\`\`\` to a row of data. We will improve our code to address each of these issues in turn, except the last one, which will be left for next week. `35:T5d2, ----- ### Sorting We can [sort an array with regular javascript](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort), but [D3 makes it so much easier](https://observablehq.com/@d3/d3-ascending)! The code can be read out loud as "given the list of \`states\`, treat each one in the list as \`a\` and compare its \`name\` with the \`name\` of the next item in the array \`b\`." [`d3.ascending()`](https://observablehq.com/@d3/d3-ascending) returns true or false, depending on if a or b is larger, and it has an inversion called `d3.descending()` that works the opposite way. The arrow operator \`=>\` is very important, and will be the topic of lots of future conversation. For now, whenever you see it, read it as "entering into the array and looking at each item one at a time". In the very near future, \`=>\` will replace \`for\` loops entirely. Notice that we are creating new Observable global variables (here `alphabeticalStates`) every time we transform our data, in accordance with [*literate programming* principles](https://observablehq.com/@observablehq/how-observable-runs). We want our code to be clear to engaged readers, and creating many variables with clear names is a very effective shortcut to more readable code — especially when there might be confusion otherwise. If you wanted to, you could sort by population instead, just by changing the key that is called in the `d3.ascending(...)` area of the code - give it a try! 36:T522,{ //svg variables const width = 800; const height = 100; //create SVG artboard const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); //simple data const exampleData = [9, 2, 7.1, 5, 0, 1, 6, 3.15, -3, 13]; //draw horizontal line svg .append("line") .attr("x1", 200) .attr("y1", height / 2) .attr("x2", 600) .attr("y2", height / 2) .attr("fill", "none") .attr("stroke", "black") .attr("stroke-width", "2"); //loop through all items in dataset for (let i = 0; i < exampleData.length; i = i + 1) { //convert data to pixels const xPosition = exampleScale(exampleData[i]); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height / 2) .attr("r", 5) .attr("fill", "orange") .attr("stroke", "black") .attr("stroke-width", "2"); //draw text for each item svg .append("text") .attr("x", xPosition) .attr("y", height / 2 - 20) .style("font-family", "courier") .style("font-size", 10) .style("text-anchor", "middle") .text(exampleData[i] + "→" + exampleScale(exampleData[i])); } //show visualization in Observable return svg.node(); }37:T52c,{ //svg variables const width = 800; const height = 100; //create SVG artboard const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); //dataset const exampleData = [1, 10, 100, 1000, 10000, 100000]; //draw horizontal line svg .append("line") .attr("x1", 200) .attr("y1", height / 2) .attr("x2", 600) .attr("y2", height / 2) .attr("fill", "none") .attr("stroke", "black") .attr("stroke-width", "2"); //loop through all items in dataset for (let i = 0; i < exampleData.length; i = i + 1) { //convert data to pixels const xPosition = exampleLogScale(exampleData[i]); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height / 2) .attr("r", 5) .attr("fill", "orange") .attr("stroke", "black") .attr("stroke-width", "2"); //draw text for each item svg .append("text") .attr("x", xPosition) .attr("y", height / 2 - 20) .style("font-family", "courier") .style("font-size", 10) .style("text-anchor", "middle") .text(exampleData[i] + "→" + Math.round(exampleLogScale(exampleData[i]))); } //show visualization in Observable return svg.node(); }38:T53f,{ //svg variables const width = 800; const height = 100; //create SVG artboard const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); //dataset const exampleData = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]; //draw horizontal line svg .append("line") .attr("x1", 200) .attr("y1", height / 2) .attr("x2", 600) .attr("y2", height / 2) .attr("fill", "none") .attr("stroke", "black") .attr("stroke-width", "2"); //loop through all items in dataset for (let i = 0; i < exampleData.length; i = i + 1) { //convert data to pixels const xPosition = exampleFlipScale(exampleData[i]); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height / 2) .attr("r", 5) .attr("fill", "orange") .attr("stroke", "black") .attr("stroke-width", "2"); //draw text for each item svg .append("text") .attr("x", xPosition) .attr("y", height / 2 - 20) .style("font-family", "courier") .style("font-size", 10) .style("text-anchor", "middle") .text( exampleData[i] + "→" + Math.round(exampleFlipScale(exampleData[i])) ); } //show visualization in Observable return svg.node(); }39:T68f,{ //svg variables const width = 800; const height = 100; //create SVG artboard const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); //create empty dataset const exampleData = []; //fill up random dataset for (let i = 0; i < 5; i = i + 1) { exampleData.push(Math.random() * 50); } //extract the min and the max of the dataset const exampleExtent = d3.extent(exampleData); //create scale to convert data to pixels const exampleScale = d3.scaleLinear().domain(exampleExtent).range([200, 600]); //draw horizontal line svg .append("line") .attr("x1", 200) .attr("y1", height / 2) .attr("x2", 600) .attr("y2", height / 2) .attr("fill", "none") .attr("stroke", "black") .attr("stroke-width", "2"); //loop through all items in dataset for (let i = 0; i < exampleData.length; i = i + 1) { //convert data to pixels const xPosition = exampleScale(exampleData[i]); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height / 2) .attr("r", 5) .attr("fill", "orange") .attr("stroke", "black") .attr("stroke-width", "2"); //draw text for each item svg .append("text") .attr("x", xPosition) .attr("y", height / 2 - 20) .style("font-family", "courier") .style("font-size", 10) .style("text-anchor", "middle") .text( exampleData[i].toFixed(1) + "→" + exampleScale(exampleData[i]).toFixed(1) ); } //show visualization in Observable return svg.node(); }3a:T693,color_scales = md`----- ### Color Scales Another kind of scale, which we ❤️ above most others, is the ability for D3 to take a number and spit out a color proportionally, in the same way we spit out pixel values proporationally with regular scales. This lets us use color *semantically* - color with meaning! These are called *color scales*, and there are a bunch of [built-in gradients](https://observablehq.com/@d3/color-schemes) (important link!) that we can sample, as well as the ability to make our own. Normally, we use the built-in color scales since they were [designed with data](https://colorbrewer2.org/#type=sequential&scheme=BuGn&n=3) to be *perceptually-uniform*. We will do a deep dive into *semantic* color choice in future weeks, as it is a complicated and important topic worthy of coversation and additional study. The names of these scales are universal across many different libraries and programming languages, and are in many ways a shared language across all data visualization practitioners. And, everyone loves *viridis*. Color scales are used just like linear or exponential scales. But, we cannot set the domain or range of a color scale — so, we need to instead contort our data to fit with what color scales expect, which is a number between 0 and 1. We call such values, defined as a position between 0 and 1, a **parameter**. So, we often need to make one scale to use with the color scales. Sigh. Change the value fed to the scale below to generate data-driven color values. Note that color scales clamp! Any number below or above the [0,1] domain of the color scale maps to the lowest or highest value in the gradient, respectively. `3b:T921,{ //svg variables const width = 800; const height = 100; //create SVG artboard const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); //simple data const exampleData = [1, 2, 3, 4, 5, 6, 7, 8, 9]; //create scale to convert data to pixels const pixelScale = d3.scaleLinear().domain([0, 10]).range([200, 600]); //create scale to convert data to number between 0 and 1 const parameterScale = d3.scaleLinear().domain([0, 10]).range([0, 1]); const scaleNames = ["viridis", "magma", "cividis"]; for (let i = 0.25; i <= 0.75; i = i + 0.25) { //draw horizontal line svg .append("line") .attr("x1", 200) .attr("y1", height * i) .attr("x2", 600) .attr("y2", height * i) .attr("fill", "none") .attr("stroke", "black") .attr("stroke-width", "2"); //draw label svg .append("text") .attr("x", 190) .attr("y", height * i) .style("font-family", "courier") .style("font-size", 10) .style("text-anchor", "end") .style("alignment-baseline", "middle") .text(scaleNames[i * 4 - 1]); } //loop through all items in dataset for (let i = 0; i < exampleData.length; i = i + 1) { //convert data to pixels const xPosition = pixelScale(exampleData[i]); //convert data to number between 0 and 1 const parameterData = parameterScale(exampleData[i]); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height * 0.25) .attr("r", 10) .attr("fill", d3.interpolateViridis(parameterData)) .attr("stroke", "black") .attr("stroke-width", "2"); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height * 0.5) .attr("r", 10) .attr("fill", d3.interpolateMagma(parameterData)) .attr("stroke", "black") .attr("stroke-width", "2"); //draw circle for each item svg .append("circle") .attr("cx", xPosition) .attr("cy", height * 0.75) .attr("r", 10) .attr("fill", d3.interpolateCividis(parameterData)) .attr("stroke", "black") .attr("stroke-width", "2"); } //show visualization in Observable return svg.node(); }3c:T4a2,md` ##### Simple Conditions Though we have not been covered conditionals formally yet, their behavior should be familiar since we have used them to limit the looping of \`for\` loops previously. Programming conditionals borrow from a [linguistic structure](https://en.wikipedia.org/wiki/Conditional_sentence) common to many languages providing alternate options, and in Javascript are introduced by the keywords \`\`if(){}, else if(){}, and else\`\`. The behavior is straightforward — each \`\`if\`\` checks whether or not a simple question is true or not. *Less than* (<), *greater than* (>), *less than or equal to* (<=), *greater than or equal to* (>=), *equal to* (==), and *not equal to* (!=) are all options for comparison. Note the *double* equal sign (==) on *equal to*, it is easy to forget! In the event of an \`if()\` condition being true, the code within the following containing braces is run. Otherwise, that code is skipped and processing advances to any number of sequential \`else if()\` blocks in order, before landing in a final catch-all \`else()\` block. Note how the code below demonstrates conditional, fallback behaviors based on the value of the slider. `3d:T729,betterCirclePlot = { //svg variables const height = 400; const margin = 20; //create SVG artboard const svg = d3.create("svg").attr("width", width).attr("height", height); //background color const background = svg .append("rect") .attr("width", width) .attr("height", height) .attr("fill", "#eeeeff"); //scales to transform data to relevant visualization values const indexToX = d3 .scaleLinear() .domain([0, alphabeticalStates.length]) .range([margin, width - margin]); const populationToRadius = d3 .scaleLinear() .domain(d3.extent(alphabeticalStates, (d) => d.population)) .range([2, (width - margin * 2) / alphabeticalStates.length]); const populationToParameter = d3 .scaleLinear() .domain(d3.extent(alphabeticalStates, (d) => d.population)) .range([0, 1]); //loop through all items in dataset for (let i = 0; i < alphabeticalStates.length; i++) { //draw circle for each item svg .append("circle") .attr("cx", indexToX(i)) .attr("cy", height / 2) .attr("r", populationToRadius(alphabeticalStates[i].population)) .attr( "fill", d3.interpolateViridis( populationToParameter(alphabeticalStates[i].population) ) ) .attr("opacity", 1); //check if i is even or odd (i%2 == 0), and assign alternating y positions const labelY = i % 2 == 0 ? height / 2 - 30 : height / 2 + 30; //draw labels svg .append("text") .attr("x", indexToX(i)) .attr("y", labelY + 4) .attr("font-size", 8) .attr("text-anchor", "middle") .attr("alignment-baseline", "center") //substring cuts certain characters from a string .text(states[i].name.substring(0, 8)); } //show visualization in Observable return svg.node(); }3e:T6d5,betterRectPlot = { //svg variables const height = 500; const margin = 20; //create SVG artboard const svg = d3.create("svg").attr("width", width).attr("height", height); //background color const background = svg .append("rect") .attr("width", width) .attr("height", height) .attr("fill", "#eeeeff"); //scales to transform data to relevant visualization values const indexToY = d3 .scaleLinear() .domain([0, alphabeticalStates.length]) .range([margin, height - margin]); const populationToWidth = d3 .scaleLinear() .domain(d3.extent(alphabeticalStates, (d) => d.population)) .range([50, 400]); const populationToParameter = d3 .scaleLinear() .domain(d3.extent(alphabeticalStates, (d) => d.population)) .range([0, 1]); //loop through all items in dataset for (let i = 0; i < alphabeticalStates.length; i++) { const rowHeight = (height - margin * 2) / states.length; //draw rect for each item svg .append("rect") .attr("x", 150) .attr("y", indexToY(i)) .attr("width", populationToWidth(alphabeticalStates[i].population)) .attr("height", height / alphabeticalStates.length) .attr( "fill", d3.interpolateMagma( populationToParameter(alphabeticalStates[i].population) ) ); //draw labels for each item svg .append("text") .attr("x", margin) .attr("y", indexToY(i) + height / alphabeticalStates.length) //the extra addition is required since text is set with its bottom-left coordinate, but rectangles are set with their top left .attr("font-size", 8) .text(states[i].name); } //show visualization in Observable return svg.node(); }3f:T66c,md` ----- ### Data Visualization Challenge : Scatter Plot! Let us create a census-based scatterplot to compare speakers of additional languages to English in American households. The census variable **group** [B16001](https://api.census.gov/data/2021/acs/acs1/groups/B16001.html) contains data on the use of many different languages across the country. Choose two languages from the [linked list of variables](https://api.census.gov/data/2021/acs/acs1/groups/B16001.html) — making sure you grab the variable codes that end in "E" for "Population **E**stimate and not "EA", "M", or "MA" which instead return "Estimation Annotation", "Margin of Error", and "Margin of Error Annotation" data respectively — and create a scatterplot comparing states. Take a look at the example visualizations and code below (which you can copy and paste into your own Observable notebooks... don't bother importing) to view how you can grab multiple census variables in one API request. Also note that the Census API might return \`null\` in the event of zero people matching the query (which importantly does not necessarily mean there are 0 people, but instead that the statistical sampling did not have enough data to report a value! We will discuss this distinction next week). A ternary operation can clean that up for us, no problem! You can make these look however you want! You will see in the screenshots below the addition of some orienting minor grid lines, as well as major axis lines around the outside. Extra goal: can you calculate the *percentage of each state's total population* that speaks each language and show that in a second plot? `40:[{"id":"a6e3f79e942dca8a103c84bfbd7c8b5d6d65460cc75910e9c53e20f054a7ab46062227fbd629d887a381bdcee38b29fa3aaba929ac9ac4fcc9cc64b75e23142b","href":"https://static.observableusercontent.com/files/a6e3f79e942dca8a103c84bfbd7c8b5d6d65460cc75910e9c53e20f054a7ab46062227fbd629d887a381bdcee38b29fa3aaba929ac9ac4fcc9cc64b75e23142b","name":"langCount.png","type":"image/png","lastModified":1663849828209.009,"size":216480},{"id":"aa327d8ae160b8a2532fb0ea842597561c36f92b1e836c6457608dfc6a44097ee8ffcda4d0a3b875f05b5a8296da7d6c9c6f9270f5e53d2bf619c1e13422eef3","href":"https://static.observableusercontent.com/files/aa327d8ae160b8a2532fb0ea842597561c36f92b1e836c6457608dfc6a44097ee8ffcda4d0a3b875f05b5a8296da7d6c9c6f9270f5e53d2bf619c1e13422eef3","name":"langPercent@1.png","type":"image/png","lastModified":1663849828209.009,"size":251739}] 2f:{"title":"Week 5 : Intro to D3 and Data Synthesis","cells":[{"id":0,"value":"title = md`\n# Week 5 : Intro to D3 and Data Synthesis\n## Cleaning Datasets, Scaling Data Points, and D3 Charting\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":964,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":969,"value":"censusData = await d3.json(\n \"https://api.census.gov/data/2024/acs/acs1?get=NAME,B01001_001E&for=state:*\"\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":987,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"dataset_intro = md`\n-----\n### Cleaning and Transforming the Dataset\nThat header row is really annoying, as are the simple arrays for each state. It would be nice if we had Javascript objects for each state. Let's transform the data to better serve our visualization work with a \\`for\\` loop that skips the header row and creates an object for each state.\n\nNote that, in previous tutorials, we used \\`i=i+1\\` to update the variable \\`i\\` between loops. We can abbreviate that operation in Javascript and many other languages to \\`\\`i++\\`\\` (but annoyingly not Python!). This shorthand \\`\\`++\\`\\` is called the **increment** operator, but is more often pronounced \"plus plus\". **Decrement** is written \\`\\`--\\`\\` and pronounced \"min min,\" working exactly the same way, but subtracting one rather than adding.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1008,"value":"states = {\n const statesData = [];\n\n //note we are starting at 1 to skip the first row!\n for (let i = 1; i < censusData.length; i++) {\n const state = censusData[i];\n statesData.push({\n name: state[0],\n population: parseInt(state[1]),\n fipsCode: state[2]\n });\n }\n return statesData;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":39,"value":"md`\nThe data here was already complete and mostly clean. All we needed to do was transform the census CSV file into javascript objects, and convert the population strings into integers. When working with other datasets, rarely will our cleaning work be this easy, but it is always important that we do data cleaning **before** any visualization work!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":47,"value":"dataset_visualization = md`\n-----\n### Dataset Visualization\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1048,"value":"md`\nAnd now, just like last week, we can make a really simple, but really *ugly and illegible* visualization.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1111,"value":"circlePlot = {\n //svg variables\n const height = 200;\n const margin = 20;\n\n //create SVG artboard\n const svg = d3.create(\"svg\").attr(\"width\", width).attr(\"height\", height);\n\n //loop through all items in dataset\n for (let i = 0; i < states.length; i++) {\n //draw circle for each item\n svg\n .append(\"circle\")\n .attr(\"cx\", i * (width / states.length) + margin)\n .attr(\"cy\", height / 2)\n .attr(\"r\", states[i].population / 1000000)\n .attr(\"fill\", \"blue\")\n .attr(\"opacity\", 0.5);\n\n //draw labels\n svg\n .append(\"text\")\n .attr(\"x\", i * (width / states.length) + margin)\n .attr(\"y\", 50)\n .attr(\"font-size\", 8)\n .attr(\"text-anchor\", \"middle\")\n .text(states[i].name);\n }\n\n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1160,"value":"md`\nHmmm. Lots of issues. Let's try rectangles instead.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1051,"value":"rectPlot = {\n //svg variables\n const height = 500;\n const margin = 20;\n\n //create SVG artboard\n const svg = d3.create(\"svg\").attr(\"width\", width).attr(\"height\", height);\n\n //loop through all items in dataset\n for (let i = 0; i < states.length; i++) {\n const rowHeight = (height - margin * 2) / states.length;\n\n //draw rect for each item\n svg\n .append(\"rect\")\n .attr(\"x\", 150)\n .attr(\"y\", i * rowHeight + margin)\n .attr(\"width\", states[i].population / 50000)\n .attr(\"height\", height / states.length)\n .attr(\"fill\", \"blue\");\n\n //draw labels\n svg\n .append(\"text\")\n .attr(\"x\", 0)\n //the extra rowHeight is required since text is set with its bottom-left coordinate, but rectangles are set with their top left\n .attr(\"y\", i * rowHeight + margin + rowHeight)\n .attr(\"font-size\", 8)\n .text(states[i].name);\n }\n\n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1164,"value":"md`\nStill ugly! But, we successfully visualized the census data!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":121,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":1178,"value":"$35","pinned":false,"hidden":false,"mode":"md"},{"id":1185,"value":"alphabeticalStates = states.sort((a, b) => d3.ascending(a.name, b.name))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1932,"value":"\n-----\n### D3 Scales\n\nConverting data units to visualization units consistently and controllably is at the core of data visualization. [D3 scales](https://github.com/d3/d3-scale) let us do that, so we never need to do manual math to convert data to pixels again!\n\nScales are functions that have an input **domain** and an output **range**, and all they do is convert numbers from one to the other. The example scale below takes numbers between 0 and 10, and returns numbers between 200 and 600. 0 becomes 200, 10 becomes 200, 5 becomes 400, and everything else *scales* proportionally.\n\nThink about the expressions we generated to convert longitude and latitude when making simple maps last week, like \"longitude plus 180 times 2 equals pixel position\". These were our scales, and d3 scales will help us generate these transformations going forward. \n","pinned":false,"hidden":false,"mode":"md"},{"id":1934,"value":"exampleScale = d3.scaleLinear().domain([0, 10]).range([200, 600])","pinned":true,"hidden":false,"mode":"ojs"},{"id":175,"value":"\nThe domain is the low and the high number that define the boundaries of the input data to be considered. The range, similarly, sets the low and the high number that define the boundaries of the data spit out. Most often, the domain is defined by a dataset's units, and the range is set to the pixels available in visualization space.\n\nPlay with the value fed to `exampleScale()` below to see how the output changes. Also adjust the values in the `domain()` and `range()` as well to see how everything changes.\n","pinned":false,"hidden":false,"mode":"md"},{"id":1938,"value":"exampleScale(5)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1936,"value":"\nNote that scales (by default) do not **clamp**. We can ask the scale above to transform -3, which is smaller than the lowest number in our domain, and the scale will return 80, which is *proportionally* smaller than the smallest number in our range.\n","pinned":false,"hidden":false,"mode":"md"},{"id":200,"value":"$36","pinned":true,"hidden":false,"mode":"ojs"},{"id":187,"value":"exotic_scales = md`\nMost often, we use linear scaling as above, which transforms numbers in one domain to a different range uniformly. Also available to us, amongst others, are logarithmic and exponential scales, which have been used throughout the COVID pandemic to communicate viral spread to middling effect. [Humans are bad at intuiting exponential growth.](https://www.pnas.org/content/117/28/16264)\n\nWe can think about these other more exotic scales as scaling each number differently, depending on its magnitude. Here, even though the numbers are spread out by different amounts from one another, they show up evenly spaced on the line.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1958,"value":"exampleLogScale = d3\n .scalePow()\n .exponent(0.01)\n .domain([1, 10000])\n .range([200, 600])","pinned":true,"hidden":false,"mode":"ojs"},{"id":254,"value":"$37","pinned":true,"hidden":false,"mode":"ojs"},{"id":377,"value":"inverting_scales = md`\nThe order of values entered as domains and ranges is meaningful. This makes it possible to flip-around datasets, like we saw possible with that confusing latitude subtraction exercise last week. Might be hard to see the difference in the code below, but the only change is in order of the items in \\`\\`.range()\\`\\`. Big and then small, rather than the usual small and then big.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1964,"value":"exampleFlipScale = d3.scaleLinear().domain([0, 10]).range([600, 200])","pinned":true,"hidden":false,"mode":"ojs"},{"id":383,"value":"$38","pinned":true,"hidden":false,"mode":"ojs"},{"id":298,"value":"statistical_tools = md`\n-----\n### Statistical Tools\nVery often, we do not know the bounding low and high numbers of a dataset — especially when ingesting dynamic or live data. For example, if we wanted to transform our census data to pixel coordinates with a scale, what should the domain be? We would need to manually find the highest and lowest population values, which is not ideal.\n\nSo, it can be helpful to extract those values programmatically. This is a very common coding pattern when working with scales, and so D3 provides [excellent statistical utilities](https://observablehq.com/@d3/d3-extent) for extracting meaningful data from a dataset. Take a look at the code below, click through its output, and run it a few times to see the numbers change.\n\nTake note specifically of \\`\\`d3.extent()\\`\\`. It gives us everything we need to setup scale domains in exactly the right format (a two item array)!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":302,"value":"{\n //create empty dataset\n const exampleData = [];\n\n //fill up random dataset\n for (let i = 0; i < 10; i = i + 1) {\n exampleData.push(Math.random() * 50);\n }\n\n const min = d3.min(exampleData);\n const max = d3.max(exampleData);\n const mean = d3.mean(exampleData);\n const extent = d3.extent(exampleData);\n\n return {\n minimum: min,\n maximum: max,\n average: mean,\n extent: extent,\n data: exampleData\n };\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":320,"value":"md`\nExtents and scales are frequently combined like this. Note that the lowest number of the dataset will transform into the lowest number of the range, so this approach automatically **clamps** the data. Transformed data will never fall outside of the range, which is often the goal (but not always!).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":277,"value":"$39","pinned":true,"hidden":false,"mode":"ojs"},{"id":401,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs"},{"id":420,"value":"{\n const parameterScale = d3.scaleLinear().domain([0, 10]).range([0, 1]);\n\n return d3.interpolateViridis(parameterScale(5));\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":457,"value":"md`\nTo clarify, the data flow is like this...\n\n\\`\\`\\`\nnumbers between a [min,max] -> \n -> regular scale with domain [min,max] and range [0,1] -> \n -> scaled numbers between [0,1] -> \n -> choice of color scale -> \n -> color values\n\\`\\`\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":450,"value":"$3b","pinned":true,"hidden":false,"mode":"ojs"},{"id":1353,"value":"conditionals = md`\n-----\n\n### Conditionals\nOne of the other issues we often encounter in data processing and visualization is *differential action*; splitting our data elements into separate cases, and handling those cases independently. For this purpose, we write [conditional control structures](https://www.javascript.com/learn/conditionals) to create branching behavior in code.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1371,"value":"$3c","pinned":false,"hidden":false,"mode":"ojs"},{"id":1385,"value":"viewof moonPhase = Inputs.range([0.0, 1.0], { label: \"Moon Phase\", value: 0.5 })","pinned":false,"hidden":false,"mode":"ojs"},{"id":1392,"value":"{\n if (moonPhase >= 0.93) {\n return \"🌕\";\n } else if (moonPhase >= 0.79) {\n return \"🌖\";\n } else if (moonPhase >= 0.65) {\n return \"🌗\";\n } else if (moonPhase >= 0.51) {\n return \"🌘\";\n } else if (moonPhase >= 0.37) {\n return \"🌑\";\n } else if (moonPhase >= 0.23) {\n return \"🌒\";\n } else if (moonPhase >= 0.09) {\n return \"🌓\";\n } else {\n return \"🌕\";\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1478,"value":"md`\nImportantly, the comparisons that we use in conditionals resolve to simple **true** or **false** values when used on their own. In this way, we can think of conditional statements as *encoded questions*.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1483,"value":"viewof booleanSlider = Inputs.range([0.0, 1.0], {\n label: \"Larger than .5?\",\n value: 0.5\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1488,"value":"booleanSlider > 0.5","pinned":true,"hidden":false,"mode":"ojs"},{"id":1375,"value":"md`\n##### Compound Conditionals\nConditionals structures can check for multiple comparisons at the same time by way of **compound conditional operators**. Two conditionals can be joined with \\`\\`||\\`\\` (pronounced \"or\"), which will exercute if *either* condition is true. Alternatively (and more usefully), \\`\\`&&\\`\\` (pronounced \"and\") requires *both* conditionals to be true for the code to run.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1446,"value":"viewof distance = Inputs.range([0.0, 1.0], { label: \"Distance to Travel\", value: 0.5 })","pinned":false,"hidden":false,"mode":"ojs"},{"id":1449,"value":"viewof time = Inputs.range([0.0, 1.0], { label: \"Time Available\", value: 0.5 })","pinned":false,"hidden":false,"mode":"ojs"},{"id":1444,"value":"{\n if (distance > 0.75 && time < 0.25) {\n return \"Recommended Transit Method: 🚀\";\n } else if (distance > 0.5 && time < 0.5) {\n return \"Recommended Transit Method: 🏎\";\n } else if (distance > 0.25 && time < 0.75) {\n return \"Recommended Transit Method: 🚲\";\n } else if (distance >= 0 && time <= 1) {\n return \"Recommended Transit Method: 🛼\";\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1428,"value":"md`\n##### Ternary Operator\n\nThis is kind of a weird one. Because writing a whole conditional structure takes lots of time, Javascript has a shortcut called the **ternary operator**, which allows us to write conditionals much more quickly. \"Ternary\" means *containing three parts*, which is a good description!\n\nTernary operators start with a conditional comparison followed by a \\`?\\` (remember that these are encoded questions!), then a value for true followed by a \\`:\\` followed by a value for false. It's easier to understand via looking at the code. \n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1505,"value":"viewof grapeAge = Inputs.range([0.0, 100], {\n label: \"How old are you?\",\n value: 18\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1503,"value":"grapeAge < 21 ? \"🍇->🧃\" : \"🍇->🍷\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":1531,"value":"md`\nThese ternary operations are perhaps most commonly encountered with a confusing arithmetic operation called a **modulus** or *modulo*, represented by a percent sign \\`%\\`. Remember when you were younger and, when doing division problems, you would have a remainder at then end? The modulo operator is like that! \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1550,"value":"viewof modulo = Inputs.range([1, 9], { label: \"9 %\", step: 1 })","pinned":false,"hidden":false,"mode":"ojs"},{"id":1545,"value":"9 % modulo","pinned":true,"hidden":false,"mode":"ojs"},{"id":1571,"value":"md`\nThe modulus operator is *very commonly* used to check if a number is even or odd via a ternary expression.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1575,"value":"viewof isItEvenOrOdd = Inputs.range([1, 9], {\n label: \"Is it even or odd?\",\n step: 1,\n value: 3\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1583,"value":"isItEvenOrOdd % 2 == 0 ? \"Even\" : \"Odd\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":522,"value":"map_with_colorscales = md`\n-----\n\n### Back to the Visualizations!\nLet's use all these techniques to improve our simple visualizations.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"$3d","pinned":true,"hidden":false,"mode":"ojs"},{"id":1304,"value":"$3e","pinned":true,"hidden":false,"mode":"ojs"},{"id":560,"value":"md`\nCool.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":601,"value":"transposing = md`\n-----\n### Beyond Boring Charts \nSo, we made prettier population charts than when we started. We cleaned-up our code with scales, sorted data, added some semantic color, and used statisitical properties to enable our charts to ingest data dynamically. That's a big deal! \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":840,"value":"next_steps = md`\n-----\n### Next Steps\nThere are some additional improvements that we will discuss in class next week, in particular replacing what is now a *very clunky* \\`\\`for\\`\\` loop structure with the much more elegant \\`\\`d3.data()\\`\\` and \\`\\`d3.enter()\\`\\` data joining structure necessary for advanced animation and interactivity. \n\nAfter that, we will soon encounter more specific data types and handling utilities such as [geographic polygons and shapefiles](https://github.com/d3/d3-geo), [bundling and hierarchical data](https://github.com/d3/d3-chord/tree/v2.0.0) structures, and [D3's time comprehensions](https://github.com/d3/d3-time-format). \n\nBut, we will certainly be pausing here for a bit, to ensure that all of this new stuff clicks after our breakneck pace so far. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1603,"value":"$3f","pinned":false,"hidden":false,"mode":"ojs"},{"id":1610,"value":"//B01001_001E is Total Population\n//B16001_105E is Arabic speaking population\n//B16001_003E is Spanish speaking populations\n\nlanguageData = await d3.json(\n \"https://api.census.gov/data/2021/acs/acs1?get=NAME,B01001_001E,B16001_105E,B16001_003E&for=state:*\"\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1619,"value":"languageStates = {\n const data = [];\n\n for (let i = 1; i < languageData.length; i++) {\n const state = languageData[i];\n data.push({\n name: state[0],\n totalPopulation: parseInt(state[1]),\n arabic: state[3] == null ? 0 : parseInt(state[2]),\n arabicPercent: state[3] == null ? 0 : parseInt(state[2]) / state[1],\n spanish: state[3] == null ? 0 : parseInt(state[3]),\n spanishPercent: state[3] == null ? 0 : parseInt(state[3]) / state[1],\n fipsCode: state[4]\n });\n }\n return data;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1796,"value":"md`As an example, here is Arabic speakers compared against Spanish speakers.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1855,"value":"FileAttachment(\"langCount.png\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":1865,"value":"md`\nAnd the same language comparison, but percentage of total population. In both plots color is set by total population.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1834,"value":"FileAttachment(\"langPercent@1.png\").image()","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W40","stdlib":"1","titling":"auto"}