2f:T416,Option2b = md`

### Option 2b: Access Arquero Data using Column Accessors Using the code \`table.get('name', row)\` for each value access feels a bit long-winded. To make the code more readable and reusable, we can instead get values using _column accessors_. Along the way, we also get a slight efficiency boost by performing column name lookups only once. The Arquero table \`getter\` method generates an accessor function that takes a row index as input and outputs the corresponding column value. The code is akin to: ~~~ js const column = table.column(columnName); return row => column.get(row); ~~~ That's all there is to it! We can define accessors for each of the columns we wish to use in our chart. This convention allows us to refer to variables in a more concise and readable fashion. Moreover, this approach makes it easier to reuse our chart design: to switch to a new dataset or different columns, we need only redefine our accessors (and our scale domains and indices), _without modifying any of the subsequent D3 code_. `30:[{"id":"81d7bd5e9551f005d7a4764e2dcb46c44f04b5918551ce19dba191a8799b498beddb5ef2da994047586fc7231749e8911c825b1967f22814a57bee58c590c86e","href":"https://static.observableusercontent.com/files/81d7bd5e9551f005d7a4764e2dcb46c44f04b5918551ce19dba191a8799b498beddb5ef2da994047586fc7231749e8911c825b1967f22814a57bee58c590c86e","name":"us-population-state-age.csv","type":"text/csv","lastModified":1672691874310.286,"size":3516}] 2d:{"title":"Arquero and D3","cells":[{"id":0,"value":"md`# Arquero and D3`","pinned":false,"hidden":false,"mode":"ojs"},{"id":68,"value":"md`In this notebook, we will look at how to use [Arquero](https://observablehq.com/@uwdata/introducing-arquero) and [D3](https://observablehq.com/@d3) together: first using Arquero to wrangle the data prior to visualization, and then considering different ways to use Arquero-backed data as input to D3.\n\nAs our running example, let's recreate a [D3 Barcode Plot](https://observablehq.com/@d3/barcode-plot) visualizing the percentage of population by age for each U.S. state, based on [U.S. Census](https://observablehq.com/@mbostock/working-with-the-census-api) data.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":37,"value":"md`

\n## Step 1: Wrangle\n\nThe initial dataset (attached as \\`us-population-state-age.csv\\`) does not arrive in a convenient format for processing and visualization. The input text has nine columns of measures, one for each age group. We would prefer a [tidier](https://en.wikipedia.org/wiki/Tidy_data) format with one column for age groups and one column for corresponding measurements. \n\nThe original Barcode Plot example uses a mix of D3 and built-in JavaScript functions to whip the data into shape:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"d3Data = {\n const data = d3.csvParse(await FileAttachment('us-population-state-age.csv').text(), d3.autoType);\n const ages = data.columns.slice(1);\n for (const d of data) d.total = d3.sum(ages, age => d[age]);\n return ages.flatMap(age => data.map(d => ({name: d.name, age, value: d[age] / d.total})));\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":41,"value":"md`Can you quickly tell what high-level transformations are being applied just by reading the code? If your answer is _\"no\"_ or _\"not exactly\"_, you are not alone!\n\nCompare that to the equivalent transforms using Arquero:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"viewof aqData = aq\n .fromCSV(await FileAttachment('us-population-state-age.csv').text())\n .fold(aq.not('name'), { as: ['age', 'value'] })\n .groupby('name')\n .derive({ value: d => d.value / op.sum(d.value) })\n .ungroup() // groups persist; ungroup to avoid downstream surprises\n .view(10)","pinned":true,"hidden":false,"mode":"ojs"},{"id":94,"value":"md`We \\`fold()\\` the data to map all nine age columns to two columns: \\`age\\` (formerly the column name) and \\`value\\`. Next, we \\`groupby()\\` the state \\`name\\` column and \\`derive()\\` normalized percentages by dividing by the group sum. We can also include a preview of the first 10 rows for good measure (and to sanity check!).`","pinned":false,"hidden":false,"mode":"ojs"},{"id":59,"value":"md`

\n## Step 2: Visualize\n\nNow that we have our data in a proper shape, let's visualize it with D3.\nWe'll first define our desired chart height and margin, to reuse in all examples below:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":116,"value":"height = 280","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"margin = ({top: 20, right: 10, bottom: 10, left: 40})","pinned":false,"hidden":false,"mode":"ojs"},{"id":150,"value":"md`Let's also define utility functions for generating _x_ and _y_ scales and the corresponding axes for a provided domain. These functions use only standard D3 methods.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":157,"value":"xscale = domain => d3.scaleLinear()\n .domain(domain)\n .range([margin.left + 10, width - margin.right])","pinned":true,"hidden":false,"mode":"ojs"},{"id":160,"value":"yscale = domain => d3.scaleBand()\n .domain(domain)\n .rangeRound([margin.top, height - margin.bottom])\n .padding(0.1)","pinned":true,"hidden":false,"mode":"ojs"},{"id":141,"value":"xaxis = (xscale) => {\n return (g) =>\n g\n .attr(\"transform\", `translate(0,${margin.top})`)\n .call(d3.axisTop(xscale).ticks(null, \"%\"))\n .call((g) =>\n g\n .selectAll(\".tick line\")\n .clone()\n .attr(\"stroke-opacity\", 0.1)\n .attr(\"y2\", height - margin.bottom - margin.top)\n )\n .call((g) => g.selectAll(\".domain\").remove());\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":143,"value":"yaxis = (yscale) => {\n return (g) =>\n g\n .attr(\"transform\", `translate(${margin.left},0)`)\n .call(d3.axisLeft(yscale))\n .call((g) =>\n g\n .selectAll(\".tick line\")\n .clone()\n .attr(\"stroke-opacity\", 0.1)\n .attr(\"x2\", width - margin.right - margin.left)\n )\n .call((g) => g.selectAll(\".domain\").remove());\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":458,"value":"Option1a = md`
\n### Option 1a: Export to Objects\n\nA common pattern in D3 is to represent data as an _array of objects_. This corresponds to a _row-oriented_ approach: each row of a data table corresponds to an object with named properties for each column. Arquero, in contrast, uses a _column-oriented_ approach where each column is stored as an array of values: rather than having a single data object per row, values for a row are distributed across the column arrays.\n\nThat said, Arquero is happy to provide data in a row-oriented format using the \\`objects()\\` method. While this requires copying all the output data, for small- to medium-sized datasets the overhead is typically not too onerous. Below we revise the original [Barcode Plot](https://observablehq.com/@d3/barcode-plot) code, using \\`data.objects()\\` to provide the input data:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":106,"value":"chart1a = {\n const d3Data = aqData.objects();\n\n const x = xscale(d3.extent(d3Data, d => d.value));\n const y = yscale(d3Data.map(d => d.age));\n\n const svg = d3.create('svg').attr('viewBox', [0, 0, width, height]);\n\n svg.append('g').call(xaxis(x));\n svg.append('g').call(yaxis(y));\n\n svg.append('g')\n .attr('fill', 'steelblue')\n .attr('stroke-width', 10)\n .attr('pointer-events', 'all')\n .selectAll('rect')\n .data(d3Data)\n .join('rect')\n .attr('x', d => x(d.value) - 0.75)\n .attr('y', d => y(d.age))\n .attr('width', 1.5)\n .attr('height', y.bandwidth())\n .append('title')\n .text(d => `${d.name}\\n${(d.value * 100).toFixed(1)}% ${d.age}`);\n\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":462,"value":"Option1b = md`
\n### Option 1b: Export Objects from an Iterator\n\nRather than call \\`objects()\\`, we can pass an Arquero table directly anywhere an [iterable](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#The_iterable_protocol) object is accepted. Below we adapt our example to (1) calculate scale domain values using Arquero and (2) pass the Arquero table directly to D3 for data binding. The effect is similar to calling \\`objects()\\`: new row objects are created for each iteration. Objects will be re-generated across iterator calls, so for cases requiring multiple invocations, call \\`objects()\\` once and reuse the resulting array!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":444,"value":"chart1b = {\n const [domains] = aqData // <-- extract first object from table iterator\n .rollup({\n x: d => [op.min(d.value), op.max(d.value)], // [min, max] domain array\n y: d => op.array_agg_distinct(d.age) // array of unique values\n });\n \n const x = xscale(domains.x);\n const y = yscale(domains.y);\n\n const svg = d3.create('svg').attr('viewBox', [0, 0, width, height]);\n\n svg.append('g').call(xaxis(x));\n svg.append('g').call(yaxis(y));\n\n svg.append('g')\n .attr('fill', 'steelblue')\n .attr('stroke-width', 10)\n .attr('pointer-events', 'all')\n .selectAll('rect')\n .data(aqData) // <-- pass aqData directly as an iterable over row objects\n .join('rect')\n .attr('x', d => x(d.value) - 0.75)\n .attr('y', d => y(d.age))\n .attr('width', 1.5)\n .attr('height', y.bandwidth())\n .append('title')\n .text(d => `${d.name}\\n${(d.value * 100).toFixed(1)}% ${d.age}`);\n\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":168,"value":"Option2a = md`
\n### Option 2a: Access Arquero Data through the Table API\n\nIf we would like to avoid the overhead of copying our data, we can tell D3 to access values directly from an Arquero table. Instead of using data objects, we can pass an array of table row \\`indices\\` to d3-selection's \\`data()\\` method. These row indices then become the bound datum (\\`d\\`) values passed to D3 methods, and we can use them to lookup values from the underlying table columns.\n\nFirst let's use Arquero to compute scale domains in a single pass, generating the domain arrays we need to create our D3 scale instances:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":246,"value":"domains = aqData.rollup({\n x: d => [op.min(d.value), op.max(d.value)], // [min, max] domain array\n y: d => op.array_agg_distinct(d.age) // array of unique values\n })\n .objects()[0] // no groups, so just pull the first result object","pinned":true,"hidden":false,"mode":"ojs"},{"id":260,"value":"md`We can generate row indices to pass to the D3 data join:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":249,"value":"indices = d3.range(0, aqData.numRows())","pinned":true,"hidden":false,"mode":"ojs"},{"id":271,"value":"md`We now plug these into our D3 code and access data with the table \\`get()\\` method:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"chart2a = {\n const x = xscale(domains.x);\n const y = yscale(domains.y);\n\n const svg = d3.create('svg').attr('viewBox', [0, 0, width, height]);\n\n svg.append('g').call(xaxis(x));\n svg.append('g').call(yaxis(y));\n\n svg.append('g')\n .attr('fill', 'steelblue')\n .attr('stroke-width', 10)\n .attr('pointer-events', 'all')\n .selectAll('rect')\n .data(indices)\n .join('rect')\n .attr(\"x\", d => x(aqData.get('value', d)) - 0.75)\n .attr(\"y\", d => y(aqData.get('age', d)))\n .attr('width', 1.5)\n .attr('height', y.bandwidth())\n .append('title')\n .text(d => `${aqData.get('name', d)}\\n${(aqData.get('value', d) * 100).toFixed(1)}% ${aqData.get('age', d)}`);\n\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":217,"value":"md`Now our visualized data is drawn straight from an Arquero table, no copying required! However, with a slight variation on the approach above, we can streamline our code _and_ make it more reusable...`","pinned":false,"hidden":false,"mode":"ojs"},{"id":221,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs"},{"id":234,"value":"xField = aqData.getter('value')","pinned":true,"hidden":false,"mode":"ojs"},{"id":236,"value":"yField = aqData.getter('age')","pinned":true,"hidden":false,"mode":"ojs"},{"id":238,"value":"labelField = aqData.getter('name')","pinned":true,"hidden":false,"mode":"ojs"},{"id":380,"value":"md`Now we update our previous example to use the column accessors:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":227,"value":"chart2b = {\n const x = xscale(domains.x);\n const y = yscale(domains.y);\n\n const svg = d3.create('svg').attr('viewBox', [0, 0, width, height]);\n\n svg.append('g').call(xaxis(x));\n svg.append('g').call(yaxis(y));\n\n svg.append('g')\n .attr('fill', 'steelblue')\n .attr('stroke-width', 10)\n .attr('pointer-events', 'all')\n .selectAll('rect')\n .data(indices)\n .join('rect')\n .attr(\"x\", d => x(xField(d)) - 0.75)\n .attr(\"y\", d => y(yField(d)))\n .attr('width', 1.5)\n .attr('height', y.bandwidth())\n .append('title')\n .text(d => `${labelField(d)}\\n${(xField(d) * 100).toFixed(1)}% ${yField(d)}`);\n\n return svg.node();\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":302,"value":"md`
\n### So which option should I use?\n\nIf you are repurposing an existing example that uses _row-oriented_ data objects, you can use [Option 1a (\\`objects\\`)](#Option1a) to rapidly plug Arquero data into the visualization. In many cases that is all you'll need to do! However, if you are writing a visualization from scratch, or using a large data set where data copying overhead is a concern, go with the _column-oriented_ representation in [Option 2b (\\`indices\\`)](#Option2b).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":102,"value":"md`

\n## Imports\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"import {aq, op} from '@uwdata/arquero'","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"d3 = require('d3')","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W30","stdlib":"1","titling":"auto"}