30:T5fa,--- ### APIs We have so far been been accustomed to work with data that can be typed into an Observable cell or attached as a file to an Observable page. Contemporary data work, however, often requires that we interact with live data stored on a remote server, which might be much larger than we could ever conveniently hold in a single file. When dealing with a topic like data marginalization, gatekeeping access to data by locking everything up in a local CSV or Excel file is so counter-productive! We need our data to be in the "cloud", and potentially available and editable by anyone! D3 helps us access remote data, whether our goals are the creation of our own dataset, or pulling data down from someone else's web repositories. When companies and organizations want to make such large datasets available, they often offer a public **API**, an *application programming interface*, to give the world, or selected individuals, access to that data. APIs are conceptually very simple: an API is composed of a structured URL pattern, a set of data description conventions, and access protocols for filtering, sorting, and manipulating or submitting data. In practice, though there are many competing attempts at [API standardization](https://swagger.io), and almost every API you encounter will offer some unique challenge to overcome. Many web services offer data (free or for a fee) via an API. Take a look at this [awesome list of free APIs](https://github.com/public-apis/public-apis) for us to reference in the future! 31:Tac6,--- ### The US Census: State-scale Data Capture The [US Census](https://en.wikipedia.org/wiki/United_States_Census) — an extraordinary resource for data scientists, policymakers, and social science researchers — is an attempt to collect information on every single person living within the boundaries of sovereign United States territory every ten years. It is called for by the Constitution of the United States, and has never been missed (even taking place during the United States Civil War and multiple pandemics). Not only is the census vital for a better understanding of the makeup and condition of the United States, but it also serves as the official mechanism for drawing congressional and state goverment district boundaries — rendering the census a vital component of the United States political mechanism. Thousands of taxpayer-supported census workers campus the country, knock on doors, collect data, and ensure as accurate a count as possible. This act is officially termed an *enumeration*. Though an effort is made to deliver a wholistic and objective count with financial punishment for those who do not complete the census form ($100/household in 2020) and a promise of total confidentiality and indemnity, participation in the census is estimated at 74%, with some populations statistically underreporting. For instance, undocumented immigrant families are less likely to participate for fear of governmental reprisal. This skew has been modeled, and serves as a statistical adjustment in some of the census datasets which do not report raw numbers. Most census datasets, however, report nothing more than pure integer counts, and contextual offsets are left up to the data modeler. Modeled on an Ancient Roman equivalent that tracked citizenry for military draft purposes (*censore* in Latin and its derivative in most romance languages means *to determine value of something*), the success of the US census has served as a model for similar efforts across the globe and supported by the United Nations. Many countries complete their own censuses, and provide them to citizens in a similar way to the United States model. Though we will be accessing the data within the census API programmatically, data from censuses through history can be explored in [this tool](https://data.census.gov/cedsci/map?q=Total%20Population&g=0200000US1,2,3,4&tid=ACSDP1Y2015.DP05&cid=DP05_0001E&vintage=2015&layer=VT_2015_020_00_PY_D1). The [2020 shortform census form](https://www2.census.gov/programs-surveys/decennial/2020/technical-documentation/questionnaires-and-instructions/questionnaires/2020-informational-questionnaire.pdf) is incredibly, well, short! ![census](https://www.commerce.gov/sites/default/files/2020-03/censusform.jpg)32:T90e,----- ### Criticisms and Limitations of the Census Historical census data, valuable and available, is tarnished by historical realities. The censuses of the early United States did not count women, people of color, or indigineous peoples, until the landmark census of 1850 — which was administered by the ahead-of-his-time sociopoltical activist and Civil War cartographer and strategist [Joseph C.G. Kennedy](https://en.wikipedia.org/wiki/Joseph_C._G._Kennedy). The census of 1880 was used as [justification for the Chinese Exclusion Act](https://en.wikipedia.org/wiki/Chinese_Exclusion_Act). The census of 1940 was used surreptitiously to identify potential Italian, German, and Japanese sympathizers, and [inter many Japanese-American citizens](https://www.washingtonpost.com/news/retropolis/wp/2018/04/03/secret-use-of-census-info-helped-send-japanese-americans-to-internment-camps-in-wwii/). These and other historical acts of oppression that still haunt the US ennumeration efforts. Because the census strives to offer consistent data over time, it often uses non-contemporary terminology and selection biases. For instance, the census still maintains binary gender definitions, counts same-sex marriages in some states and not others, and mixes up racial and ethic identity. Counting groups, inherently, requires the drawing of lines between people — and that is often done by the census in unenlighted, conservative ways with *consistency* as the ultimate goal. Moreover, because of the role of the census in determining House of Representative seat counts as well as significant federal funding, census data collection has long been politicized. The 2020 census data was likely [poisoned](https://www.amacad.org/news/2020-census-challenges-implications) by the attempted [inclusion of a citizenship question](https://www.commoncause.org/democracy-wire/2020-census-controversies/) (leading to the fear that many undocumented immigrants would not complete the form), and despite its importance to political process and policymaking has been defunded by both political paries. Additionally, contemporary gross population trends are much more dynamic than the *every 10 year* original model designed for the on-horseback census worker. As a result, the census data collection efforts have needed to expand.33:Tb5e,----- ### The Decennial Census and the American Community Survey The limitations of a short census questionnaire caused [many issues](https://en.wikipedia.org/wiki/United_States_census#Controversy) to the young US government, but it was not till 1970 that the census form was meaningfully expanded. 90% of households received the *shortform* census questionnaire, hewing closely to the historical censuses, but [15% of randomly selected households](https://www.nap.edu/read/4805/chapter/13) instead received a *longform* questionnaire, which nearly quadrupled the number of questions asked. This split tabluation process has continued since. Since the early aughts, the 10 year cadence of the census — clearly an outdated model in an era of constant data production and collection — has been supported by a motley assortment of supplemental data gathering efforts. One of the more successful attempts has been the [American Community Survey](https://www.census.gov/data/developers/data-sets/acs-5year.html), which aims to provide census-level details of a statistical sampling of the full American population at a higher frequency for large population centers. The ACS occurs on an ongoing basis through the calendar year with a (relatively) small sample size (population centers of 65,000 individuals or greater), and is also tallied every five years with a larger sampling (population centers of 20,000 individuals or greater). The ACS tracks around [60,000 *variables*](https://api.census.gov/data/2023/acs/acs5/variables.html) across thousands of [*geographies*](https://api.census.gov/data/2023/acs/acs5/geography.html), and provides projections and estimates for another 20,000 synthetic* topic variables*. The data is voluminous, to say the least. Many of these indicators are aimed at constructing a picture of the United States' identity makeup — gender, race, ethnicity, derivation, age. Others are financial — income, employment status, employment industry, labor force participation. Further indicators support a better understanding of the American household — children/family, household size, number of rooms, rent/mortgage cost, frequency of housing changes, languages spoken at home. A large proportion of these variables have been tracked in the same way over many years, allowing data scientists and visualizers to trace and understand the increasingly tumultuous social dynamics of the United States *longitudinally*. In 2000, [the *long form census* was replaced by the equivalent questionnaire of the ACS](https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020_ch09.pdf). Since 2010, the ACS and the decenniel census have become increasingly tied together — with the Decennial Census data variables mapped as a subset of the ACS's much larger statistical pool. For this course, we can treat the two interchangeably in most cases. 34:T4ec,A few notes... - The data is `await`ed, because it takes time to download data from any other website to Observable. - We are expecting the data in the **json** format, which is an abbreviation for "Javascript Object Notation", just an array of JS objects with a few other limitations and features. - The data is from the [single yearly American Community Survey (ACS) API](https://www.census.gov/data/developers/data-sets/acs-1year.html). - The data is dated to the year `2023` - The data we are requesting includes the [census variable](https://api.census.gov/data/2023/acs/acs1/variables.html) `B01001_001E` (the total population of a geographic area) and the `NAME` for all (`*`) of the states, our geography. For more complex queries of the US Census API, we would need a census key. If you [request one](https://api.census.gov/data/key_signup.html) (yes, this weirdly basic page is a US federal website!), it is important that you not publish it on public websites to prevent abuse. We will spend more time on this API query structure on this page and in the future, but if you are curious, you can explore how to access data with more specificity by browsing the [examples](https://api.census.gov/data/2023/acs/acs1.html) provided by the US government.35:T682,md` Take a look at the resulting data that comes back from the US government above... notice anything strange? For one, the data is clearly structured as a CSV file with a header row, and not as a Javascript object. That's annoying. The first item in the returned array is the header row, and each state after is an array. But we have 53 items, and not the 51 we would expect (50 states + 1 header row). This is because, for census purposes, Washington DC and Puerto Rico are considered states. Each state array has 3 data points. The \`name\` of the state, code \`B01001_001E\` which is the total population of the state (as a string!), and finally a \`state\` code, which we did not ask for, representing the [FIPS code](https://www.nrcs.usda.gov/wps/portal/nrcs/detail/?cid=nrcs143_013696) for the state. These codes are very useful for aligning data between different sources. A quick note on that \`await\` keyword, which allows for control over the [**asynchronous**](https://nikgrozev.com/2017/10/01/async-await/) nature of contemporary Javascript. Back in the bad old days of web browsers being attached to slow computer processors and inconsistent dial-up network connections, it was decided that web browsers should run Javascript *as quick as possible*. So, if we tell a web browser to go get a file, it will go do that *and then continue running your code before the download completes*. For this reason, we cannot trust that a file is fully loaded unless we explicitly tell Javascript to wait for the download to complete. This is an annoying "feature", of Javascript, and will often give us headaches when dealing with external APIs and datasets. `36:T726,----- ### Data Visualization Challenge : ACS Data Browser! Using the techniques introduced on this page, create a data browser and visualization for a set of census codes and geographies of your choice. To get started, open a few pages in tabs for reference... - [Census geographies](https://api.census.gov/data/2023/acs/acs1/cprofile/geography.html) - [ACS groups](https://api.census.gov/data/2023/acs/acs1/groups.html) - [Examples](https://api.census.gov/data/2020/acs/acs5/examples.html) Then let's build a data browser! 1. Identify an ACS group you are interested in, get a sense for what data is available, and browse through its specific codes after clicking on "selected variables". You can of course explore multiple groups in the same API call and visualization! 2. Choose a few codes that might be cool to build a dynamic visualization around. Remember, codes ending in "E" actually have data. Almost all codes are projected population counts, but a few are not (the documentation always makes this clear). 3. Create some inputs based on the options available from the ACS group. 4. Construct your ACS query. 5. Clean-up the data with the techniques above. Then, build a visualization to showcase your data! Choose whatever you want to build and try to build it, so long as you can construct it with SVG shapes. If you are not feeling inspired, check out [datavizproject](https://datavizproject.com/) for some ideas. Choose something simple to start, there will be plenty of time for more advanced visualization types later on in the semester. Bar charts, butterfly charts, dumbbell charts, bump charts, heatmaps, proportional area charts, bubble charts, box plots, nested area proportional chart, dot plots, scatter plots, dot charts, strip plots, etc. are all possible with just SVG rectangles and circles! 37:Tb29,halfCircles = { //set up for our grid of half-circles let rows = 8; let columns = 7; //this keeps the SVG artboard consistent with our row:col ratio const height = width * (rows / columns); //size of circle for each state const scale = 30; //create SVG artboard const svg = d3.create("svg").attr("width", width).attr("height", height); //background color svg .append("rect") .attr("width", width) .attr("height", height) .attr("fill", "#f0f5f9"); //This is a nested for loop //For every loop of y, we loop through all the columns of x for (let y = 0; y < rows; y++) { for (let x = 0; x < columns; x++) { //determine which item we are currently operating on in the grid //which row we're on (y) multiplied by number of columns, plus the current column count (x) let i = y * columns + x; //for convenience let state = transitComparisonData[i]; //check if we are out of states to draw if (i > transitComparisonData.length - 1) { //kill the loop break; } //calculate the percentage difference between car-based on non-car-based transit let ratio = state.otherTransit / state.driving; //find center point and add margin let xCenter = (width / columns) * x + width / (columns * 2); let yCenter = (height / rows) * y + height / (rows * 2); //add half circle for drivers with SVG arc svg .append("path") .attr( "d", "M " + xCenter + " " + yCenter + " a " + scale + " " + scale + " 0 0 0 " + scale * 2 + " 0" ) //arcs are drawn from their sides, not centers, so we shift them back by their radius .attr("transform", "translate(" + -scale + ",0)") .attr("fill", "#6d819c"); //add half circle for non-drivers with SVG arc svg .append("path") .attr( "d", "M " + xCenter + " " + yCenter + " a " + scale * ratio + " " + scale * ratio + " 0 0 1 " + scale * ratio * 2 + " 0" ) //arcs are drawn from their sides, not centers, so we shift them back by their radius .attr("transform", "translate(" + -scale * ratio + ",0)") .attr("fill", "#75D701"); //state labels svg .append("text") .attr("x", xCenter) .attr("y", yCenter + scale + 20) .attr("font-size", 12) .attr("text-anchor", "middle") .attr("font-family", "Monaco") .attr("fill", "#274c5e") .text(stateLUT[state.name]); } } //show visualization in Observable return svg.node(); }38:[{"id":"a6e3f79e942dca8a103c84bfbd7c8b5d6d65460cc75910e9c53e20f054a7ab46062227fbd629d887a381bdcee38b29fa3aaba929ac9ac4fcc9cc64b75e23142b","href":"https://static.observableusercontent.com/files/a6e3f79e942dca8a103c84bfbd7c8b5d6d65460cc75910e9c53e20f054a7ab46062227fbd629d887a381bdcee38b29fa3aaba929ac9ac4fcc9cc64b75e23142b","name":"langCount.png","type":"image/png","lastModified":1726682248926.741,"size":216480},{"id":"aa327d8ae160b8a2532fb0ea842597561c36f92b1e836c6457608dfc6a44097ee8ffcda4d0a3b875f05b5a8296da7d6c9c6f9270f5e53d2bf619c1e13422eef3","href":"https://static.observableusercontent.com/files/aa327d8ae160b8a2532fb0ea842597561c36f92b1e836c6457608dfc6a44097ee8ffcda4d0a3b875f05b5a8296da7d6c9c6f9270f5e53d2bf619c1e13422eef3","name":"langPercent@1.png","type":"image/png","lastModified":1726682248926.741,"size":251739}] 2e:{"title":"Week 5 : Data Access and APIs","cells":[{"id":0,"value":"\n# Week 5 : Data Access and APIs\n## APIs, Cleaning Datasets, Statistical Tools\n","pinned":false,"hidden":false,"mode":"md"},{"id":964,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":2330,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":2348,"value":"The questions from each and every census years are [browsable](https://www.census.gov/history/www/through_the_decades/index_of_questions/), maintained by the Department of Commerce, which clarifies which questions are added or changed decade-to-decade. Scans of the forms themselves are also [logged](https://www.census.gov/history/www/through_the_decades/questionnaires/1970_1.html), which is a tremendous artifact of historical communication design evolution. ","pinned":false,"hidden":false,"mode":"md"},{"id":2334,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":2350,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":2372,"value":"---\n### ACS Data\n\nLet's take a look at how we can pull data from the ACS! In order to do so, we need to decide a few pieces of information\n\n- The **dataset**, like the ACS or Decennial Census\n- The **vintage**, like the ACS 1 year projections are the more accurate but less contemporary ACS 5 year projections\n- The **geography** of interest, like states or counties\n- The **code** for the desired population sample, like total population or female population over age 65 with a college degree. There are 65,000+ options!\n- The **year** of sampling. Data is currently available from 2006 to 2023, though not all data points are available for the whole timespan)\n- Optionally, The **geographic limitations**, like limiting a query for counties to only counties *in Illinois*\n\nAll of this gets composed into a url, and D3 can go and get the data for for us if we format it correctly.\n\n`https://api.census.gov/data/YEAR/DATASET/VINTAGE?get=CODE&for=GEOGRAPHY&in=LIMITATION`","pinned":false,"hidden":false,"mode":"md"},{"id":969,"value":"censusData = await d3.json(\n \"https://api.census.gov/data/2023/acs/acs1?get=NAME,B01001_001E&for=state:*\"\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2403,"value":"Visit the same link, and you'll see the data Observable grabs in your browser.","pinned":false,"hidden":false,"mode":"html"},{"id":2038,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":2379,"value":"censusData","pinned":true,"hidden":false,"mode":"ojs"},{"id":987,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"\n-----\n### Cleaning and Transforming the Dataset\nThat header row is really annoying, as are the simple arrays for each state. It would be nice if we had Javascript objects for each state instead. Let's transform the data to better serve our visualization work with a `for` loop that skips the header row and creates an object for each state.\n\nNote that, last week, we used both `i=i+1` and `i++` to update the looping variable `i` between loops. The latter is much more common in Javascript and many other languages, but is annoyingly not available in Python!. This shorthand `++` is called the **increment** operator, but is more often pronounced \"plus plus\". **Decrement** is written `--` and pronounced \"min min,\" working exactly the same way, but subtracting one rather than adding one.","pinned":false,"hidden":false,"mode":"md"},{"id":2240,"value":"As a reminder of what the data looks like before we clean it up...","pinned":false,"hidden":false,"mode":"md"},{"id":2238,"value":"censusData","pinned":true,"hidden":false,"mode":"ojs"},{"id":2242,"value":"And now, on to the cleaning and restructuring! The data here was already complete and mostly clean. All we needed to do was transform the census CSV file into javascript objects, and convert the population strings into integers. When working with other datasets, rarely will our cleaning work be this easy, but it is always important that we do data cleaning **before** any visualization work!\n","pinned":false,"hidden":false,"mode":"md"},{"id":1008,"value":"states = {\n const statesData = [];\n\n //note we are starting at 1 to skip the first row!\n for (let i = 1; i < censusData.length; i++) {\n const state = censusData[i];\n statesData.push({\n name: state[0],\n population: parseInt(state[1]),\n fipsCode: parseInt(state[2])\n });\n }\n return statesData;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":47,"value":"\n-----\n### Dataset Visualization\nAnd now, just like in previous weeks like when we made our \"Hello World maps\", we can make a very simple, but also very *ugly and illegible* visualization.","pinned":false,"hidden":false,"mode":"md"},{"id":2258,"value":"circlePlot = {\n //svg variables\n const height = 200;\n const margin = 20;\n\n //create SVG artboard\n const svg = d3.create(\"svg\").attr(\"width\", width).attr(\"height\", height);\n\n //loop through all items in dataset\n for (let i = 0; i < states.length; i++) {\n //draw circle for each item\n svg\n .append(\"circle\")\n .attr(\"cx\", i * (width / states.length) + margin)\n .attr(\"cy\", height / 2)\n .attr(\"r\", states[i].population / 1000000)\n .attr(\"fill\", \"blue\")\n .attr(\"opacity\", 0.5);\n\n //draw labels\n svg\n .append(\"text\")\n .attr(\"x\", i * (width / states.length) + margin)\n .attr(\"y\", 50)\n .attr(\"font-size\", 8)\n .attr(\"text-anchor\", \"middle\")\n .text(states[i].name);\n }\n\n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1160,"value":"\nHmmm. Lots of issues. Let's try vertical alignment (so that text does not overlap) and use rectangles instead.\n","pinned":false,"hidden":false,"mode":"md"},{"id":1051,"value":"rectPlot = {\n //svg variables\n const height = 500;\n const margin = 20;\n\n //create SVG artboard\n const svg = d3.create(\"svg\").attr(\"width\", width).attr(\"height\", height);\n\n //loop through all items in dataset\n for (let i = 0; i < states.length; i++) {\n const rowHeight = (height - margin * 2) / states.length;\n\n //draw rect for each item\n svg\n .append(\"rect\")\n .attr(\"x\", 150)\n .attr(\"y\", i * rowHeight + margin)\n .attr(\"width\", (width * states[i].population) / 100000000) //The big number here was determined just by adding zeroes!\n .attr(\"height\", height / states.length)\n .attr(\"fill\", \"blue\");\n\n //draw labels\n svg\n .append(\"text\")\n .attr(\"x\", 0)\n //the extra rowHeight is required since text is set with its bottom-left coordinate, but rectangles are set with their top left\n .attr(\"y\", i * rowHeight + margin + rowHeight)\n .attr(\"font-size\", 8)\n .text(states[i].name);\n }\n\n //show visualization in Observable\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1164,"value":"\nStill ugly, but better, and we successfully visualized the census data!","pinned":false,"hidden":false,"mode":"md"},{"id":2246,"value":"---\n### Controlling the Census Query\nWouldn't it be cool if we could access different kinds of information on the census like a data dashboard would let us do? Remember to click the \">\" to the left of inputs to see how they are built.\n","pinned":false,"hidden":false,"mode":"md"},{"id":2321,"value":"For the region of interest let's create an array of options. Read more about [ACS geographies](https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_geography_handbook_2020_ch01.pdf). Note that, to use many of these options, you'll often replace any space in the name of the geography with \"%20\", its HTML encoding alternative.\n\n\"congressional district\" => \"congressional%20district\"","pinned":false,"hidden":false,"mode":"md"},{"id":2530,"value":"geographies = [\"county\", \"congressional%20district\", \"place\"]","pinned":true,"hidden":false,"mode":"ojs"},{"id":2439,"value":"For the datacodes, let's look at [ACS group B08101](https://api.census.gov/data/2023/acs/acs1/groups/B08101.html), \"Means of Transportation to Work by Age.\" We will use the data codes for the total estimate for each method of transit, rather than the age-based segments. ","pinned":false,"hidden":false,"mode":"md"},{"id":2542,"value":"transitOptions = [\n \"Total Population\",\n \"Drove alone\",\n \"Carpooled\",\n \"Public Transit\",\n \"Walked\",\n \"Taxi, Bikes, Other\",\n \"Worked from Home\"\n]","pinned":false,"hidden":false,"mode":"ojs"},{"id":2475,"value":"Now, let's create an object that relates our human-readable transit methods to ACS codes. We can treat this object as a [look up table](https://en.wikipedia.org/wiki/Lookup_table) to convert our list of options to the census code. Note that we are only using the census codes ending in \"E\", as these are the population **E**stimates. Codes that end in \"EA\" are **E**stimate **A**nnotations, which can be helpful to better understand definitions and protocols but don't include data. Codes that end in \"M\" are **M**argin of Error. And codes ending in \"MA\" are **M**argin of error **A**nnotations, which sometimes help describe how to the margin of error was calculated.","pinned":false,"hidden":false,"mode":"md"},{"id":2311,"value":"dataCodes = ({\n \"Total Population\": \"B08101_001E\",\n \"Drove alone\": \"B08101_009E\",\n Carpooled: \"B08101_017E\",\n \"Public Transit\": \"B08101_025E\",\n Walked: \"B08101_033E\",\n \"Taxi, Bikes, Other\": \"B08101_041E\",\n \"Worked from Home\": \"B08101_049E\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2486,"value":"We can use this object like so...","pinned":false,"hidden":false,"mode":"md"},{"id":2489,"value":"dataCodes.Walked","pinned":true,"hidden":false,"mode":"ojs"},{"id":2492,"value":"But what if we want to use one of our transit methods with spaces in them... Javascript can't deal with it! (The cell below is supposed to not work.)","pinned":false,"hidden":false,"mode":"md"},{"id":2496,"value":"dataCodes.Drove alone","pinned":true,"hidden":false,"mode":"ojs"},{"id":2498,"value":"Instead, we need to use a different way of accessing keys in objects.","pinned":false,"hidden":false,"mode":"md"},{"id":2500,"value":"dataCodes[\"Drove alone\"]","pinned":true,"hidden":false,"mode":"ojs"},{"id":2502,"value":"This is also how we use a variable name as a key.","pinned":false,"hidden":false,"mode":"md"},{"id":2509,"value":"{\n const transitOption = \"Public Transit\";\n return dataCodes[transitOption];\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2564,"value":"---\n### Building Our Data Dashboard","pinned":false,"hidden":false,"mode":"md"},{"id":2511,"value":"Putting all this together, with some Inputs like last week! Note how the URL changes when you select different options.","pinned":false,"hidden":false,"mode":"md"},{"id":2268,"value":"viewof geography = Inputs.select(geographies, {\n value: \"place\",\n label: \"Geography\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2308,"value":"viewof dataChoice = Inputs.radio(\n transitOptions,\n { value: \"Walked\", label: \"Transit Method\" }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2681,"value":"viewof year = Inputs.select([\"2021\", \"2022\", \"2023\"], { label: \"Year\" })","pinned":false,"hidden":false,"mode":"ojs"},{"id":2287,"value":"acsUrl = \"https://api.census.gov/data/\" +\n year +\n \"/acs/acs1?get=NAME,\" +\n dataCodes[dataChoice] +\n \"&for=\" +\n geography +\n \"&in=state:17\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":2294,"value":"Visit this census query in your web browser.","pinned":false,"hidden":false,"mode":"html"},{"id":2265,"value":"rawTransitPopData = await d3.json(acsUrl)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2574,"value":"And we can clean this up just like before.","pinned":false,"hidden":false,"mode":"md"},{"id":2572,"value":"cleanTransitPopData = {\n const geoData = [];\n\n //note we are starting at 1 to skip the first row!\n for (let i = 1; i < rawTransitPopData.length; i++) {\n const geo = rawTransitPopData[i];\n geoData.push({\n name: geo[0],\n count: parseInt(geo[1]) ? parseInt(geo[1]) : null, //this ternary checks for \"null\", which ACS uses\n geoLimit: parseInt(geo[2]),\n geoCode: parseInt(geo[3])\n });\n }\n return geoData;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":298,"value":"\n-----\n### Statistical Tools\nVery often, we do not know the bounding low and high numbers of a dataset — especially when ingesting dynamic or live data. For example, if we wanted to transform our census data to pixel coordinates, what should the minimum and maximum be? We would need to manually find the highest and lowest population values, which is not ideal, and sometimes not even possible with dynamic data. And, we often want to work with other central characteristics of datasets like means, medians, and modes.","pinned":false,"hidden":false,"mode":"md"},{"id":2587,"value":"d3.min([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2592,"value":"d3.median([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2599,"value":"d3.mode([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2601,"value":"d3.mean([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2603,"value":"d3.max([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2630,"value":"d3.extent([0, 1, 2, 3, 4, 5, 5, 5, 6, 7])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2606,"value":"Take a look at the code below, click through its output, and run it a few times to see the numbers change.\n","pinned":false,"hidden":false,"mode":"md"},{"id":302,"value":"{\n //create empty dataset\n const exampleData = [];\n\n //fill up random dataset\n for (let i = 0; i < 10; i = i + 1) {\n exampleData.push(Math.random() * 100);\n }\n\n const min = d3.min(exampleData);\n const max = d3.max(exampleData);\n const mean = d3.mean(exampleData);\n const extent = d3.extent(exampleData);\n\n return {\n minimum: min,\n maximum: max,\n average: mean,\n extent: extent,\n data: exampleData\n };\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2634,"value":"You can also use an **arrow function** with these tools, to access a particular key in an object, as well as do other cool things. We'll learn all about these next week! The code below looks through the `states` array from above, and identifying each element as `s`, calculates the `mean` of their populations.","pinned":false,"hidden":false,"mode":"md"},{"id":2638,"value":"d3.mean(states, (s) => s.population)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1603,"value":"$36","pinned":false,"hidden":false,"mode":"md"},{"id":2712,"value":"-----\n### Some tips and an example!\n\nVery often, we need to do math on ACS data to make it more understandable. For example, if we wanted to compare in a [half-circle chart](https://datavizproject.com/data-type/proportional-area-chart-half-circle/) how many car-based commuters there are versus all other transit methods per state, we would need to do some work on the ACS data. We need to add together the people who drove and carpooled to get our complete car transit population, and also add together all the other codes to get our other transit population. ","pinned":false,"hidden":false,"mode":"md"},{"id":2717,"value":"drivingCodes = \"B08101_009E,B08101_017E\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":2720,"value":"otherTransitCodes = \"B08101_025E,B08101_033E,B08101_041E,B08101_049E\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":3104,"value":"Construct the ACS query, pull configuration from inputs (limited to keep this example short, do more!), and access data...","pinned":false,"hidden":false,"mode":"md"},{"id":3121,"value":"viewof transitYear = Inputs.radio([\"2021\", \"2022\", \"2023\"], {\n value: \"2023\",\n label: \"Year\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2710,"value":"transitPopUrl = \"https://api.census.gov/data/\" +\n transitYear +\n \"/acs/acs1?get=NAME,\" +\n drivingCodes +\n \",\" +\n otherTransitCodes +\n \"&for=state:*\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":2735,"value":"rawTransitData = await d3.json(transitPopUrl)","pinned":true,"hidden":false,"mode":"ojs"},{"id":3106,"value":"Clean up the data, and do the summing described above...","pinned":false,"hidden":false,"mode":"md"},{"id":2742,"value":"transitComparisonData = {\n const transitData = [];\n\n //note we are starting at 1 to skip the first row!\n for (let i = 1; i < rawTransitData.length; i++) {\n const state = rawTransitData[i];\n transitData.push({\n name: state[0],\n driving: parseInt(state[1]) + parseInt(state[2]),\n otherTransit:\n parseInt(state[3]) +\n parseInt(state[4]) +\n parseInt(state[5]) +\n parseInt(state[6]),\n fipsCode: parseInt(state[7])\n });\n }\n return transitData;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":3108,"value":"And time to draw! This uses some fancier techniques than we've discussed before. Don't stress about the anything that looks unfamiliar, just wanted to give a sense of what is possible — and check the comments! Also view this awesome explorable explainer on the [SVG Arc element](https://www.nan.fyi/svg-paths/arcs), which is used to draw these half-circles. No inputs were used to keep this example self-contained, but it would be cool to control colors, grid dimensions, label styles...\n\nThey look like wheels!","pinned":false,"hidden":false,"mode":"md"},{"id":2761,"value":"$37","pinned":true,"hidden":false,"mode":"ojs"},{"id":3031,"value":"The abbreviations above are pulled from this look up table.","pinned":false,"hidden":false,"mode":"md"},{"id":2987,"value":"stateLUT = ({\n Alabama: \"AL\",\n Alaska: \"AK\",\n Arizona: \"AZ\",\n Arkansas: \"AR\",\n California: \"CA\",\n Colorado: \"CO\",\n Connecticut: \"CT\",\n Delaware: \"DE\",\n \"District of Columbia\": \"DC\",\n Florida: \"FL\",\n Georgia: \"GA\",\n Hawaii: \"HI\",\n Idaho: \"ID\",\n Illinois: \"IL\",\n Indiana: \"IN\",\n Iowa: \"IA\",\n Kansas: \"KS\",\n Kentucky: \"KY\",\n Louisiana: \"LA\",\n Maine: \"ME\",\n Montana: \"MT\",\n Nebraska: \"NE\",\n Nevada: \"NV\",\n \"New Hampshire\": \"NH\",\n \"New Jersey\": \"NJ\",\n \"New Mexico\": \"NM\",\n \"New York\": \"NY\",\n \"North Carolina\": \"NC\",\n \"North Dakota\": \"ND\",\n Ohio: \"OH\",\n Oklahoma: \"OK\",\n Oregon: \"OR\",\n Maryland: \"MD\",\n Massachusetts: \"MA\",\n Michigan: \"MI\",\n Minnesota: \"MN\",\n Mississippi: \"MS\",\n Missouri: \"MO\",\n Pennsylvania: \"PA\",\n \"Puerto Rico\": \"PR\",\n \"Rhode Island\": \"RI\",\n \"South Carolina\": \"SC\",\n \"South Dakota\": \"SD\",\n Tennessee: \"TN\",\n Texas: \"TX\",\n Utah: \"UT\",\n Vermont: \"VT\",\n Virginia: \"VA\",\n Washington: \"WA\",\n \"West Virginia\": \"WV\",\n Wisconsin: \"WI\",\n Wyoming: \"WY\"\n})","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W38","stdlib":"1","titling":"auto"}