32:T402,md`[*Introducing Arquero*](https://observablehq.com/@uwdata/introducing-arquero) demonstrates the use of the [*op.corr*](https://uwdata.github.io/arquero/api/op#corr) aggregate function to determine the [product-moment correlation](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) between two variables in a table. The notebook's example shows the correlation operation being invoked manually for two variables:

...
beers
  .rollup({
   ... 
    corr_abv_ibu: d => op.corr(d.abv, d.ibu),
   ...
  })
  .view()
...

This notebook implements a general, reusable pattern to calculate correlations for all measures in a dataset. To illustrate the pattern, the 30 stocks in the [Dow Jones Industrial Average (DJIA)](https://en.wikipedia.org/wiki/Dow_Jones_Industrial_Average) will be analyzed to generate this notebook's [heatmap](https://en.wikipedia.org/wiki/Heat_map) with [Vega-Lite](https://observablehq.com/@uwdata/introduction-to-vega-lite).`33:T896,md`## Usage To use *correlationAnalysis* against your data in the Observable environment: ~~~js import {correlationAnalysis} with {yourData as data} from "@mariodelgadosr/dow-jones-industrial-average-correlation-analysis-with-ar" ~~~ Where: ***yourData*** is an array of standard JavaScript objects relative to your notebook: * Each object is interpreted as a table row with object keys serving as column names; * Each row has at least two measures; * Eg: [{[dimension1: value...], measure1: value, measure2: value...}, ...] Then reference the ***correlationAnalysis*** object in an Observable Cell. ***New: inpsired by [@jurb](https://observablehq.com/@jurb)'s [fork](https://observablehq.com/@jurb/dow-jones-industrial-average-correlation-analysis-with-ar) of this notebook*** (See: [Sankey Quick Start App](https://observablehq.com/@mariodelgadosr/sankey-quick-start-app?collection=@mariodelgadosr/sankey-visualization) for similar implementation of the one-line application technique) : To reference the complete Vega-lite heatmap (*correlationHeatMap*) against your data in the Observable environment: ~~~js import {correlationHeatMap} with {yourData as data, [yourMetricName as metricName, yourTitle as title, yourLookupTable as lookupTable ]} from "@mariodelgadosr/dow-jones-industrial-average-correlation-analysis-with-ar" ~~~ Where: ***yourData*** is an array of standard JavaScript objects relative to your notebook: * Each object is interpreted as a table row with object keys serving as column names; * Each row has at least two measures; * Eg: [{[dimension1: value...], measure1: value, measure2: value...}, ...] ***yourMetricName*** is an optional string value that will be displayed for the correlated metrics in the heat map ***yourTitle*** is an optional string value that will be displayed as the heat map's title. ***yourLookupTable*** is an optional array of objects used to translate the dimension value name in yourData. The first property of the object is the lookup table's key and the second property is the lookup table's field value. See *stockCompany* in this notebook for an example. `34:T657,stockCompany = [ { Stock: "AAPL", company: "Apple Inc." }, { Stock: "AXP", company: "The American Express Company" }, { Stock: "BA", company: "The Boeing Company" }, { Stock: "CAT", company: "Caterpillar Inc." }, { Stock: "CSCO", company: "Cisco Systems, Inc." }, { Stock: "CVX", company: "Chevron Corporation" }, { Stock: "DIS", company: "The Walt Disney Company" }, { Stock: "DOW", company: "DowDuPont Inc." }, { Stock: "GS", company: "The Goldman Sachs Group, Inc." }, { Stock: "HD", company: "The Home Depot Inc." }, { Stock: "IBM", company: "International Business Machines Corporation" }, { Stock: "INTC", company: "Intel Corporation" }, { Stock: "JNJ", company: "Johnson & Johnson" }, { Stock: "JPM", company: "JPMorgan Chase & Co." }, { Stock: "KO", company: "The Coca-Cola Company" }, { Stock: "MCD", company: "McDonald’s Corporation" }, { Stock: "MMM", company: "The 3M Company" }, { Stock: "MRK", company: "Merck & Company, Inc." }, { Stock: "MSFT", company: "Microsoft Corporation" }, { Stock: "NKE", company: "Nike, Inc." }, { Stock: "PFE", company: "Pfizer Inc." }, { Stock: "PG", company: "Proctor & Gamble Co." }, { Stock: "TRV", company: "The Travelers Companies, Inc." }, { Stock: "UNH", company: "UnitedHealth Group, Inc." }, { Stock: "UTX", company: "United Technologies Corporation" }, { Stock: "V", company: "Visa Inc." }, { Stock: "VZ", company: "Verizon Communications Inc." }, { Stock: "WBA", company: "Walgreens Boots Alliance, Inc." }, { Stock: "WMT", company: "Walmart Inc." }, { Stock: "XOM", company: "Exxon Mobil Corporation" } ]35:T7d7,heatMap = { // Concatenate correlationAnalysis to itself, switching metric_1/metric_2 for the heatmap. // https://uwdata.github.io/arquero/api/verbs#concat // https://uwdata.github.io/arquero/api/verbs#derive const concatenation = correlationAnalysis.concat( correlationAnalysis.derive({ metric_2: ({ metric_1 }) => metric_1, metric_1: ({ metric_2 }) => metric_2 }) ); let thisMetricName = metricName; let thisTitle = title; let thisLookupTable = lookupTable; if (data._fromNotebook) { thisMetricName == "DJIA Stock"; thisTitle = "DJIA 2020 Correlation Analysis"; thisLookupTable = stockCompany; } // if thisMetricName = thisMetricName || "Metric"; thisTitle = thisTitle || "Metric Correlation Analysis"; thisLookupTable = thisLookupTable || [ ...new Map( [] .concat( correlationAnalysis._data.metric_1.data, correlationAnalysis._data.metric_2.data ) .map((d) => [d, d]) ).keys() ].map((d) => ({ metric: d, field: d })); const lookupFields = Object.keys(thisLookupTable[0]); const lookup = vl .lookupData(thisLookupTable) .key(lookupFields[0]) .fields([lookupFields[1]]); return vl .markBar() .title({ text: thisTitle, subtitle: correlationsRadioButtons.subtitle }) .data(concatenation) .transform( vl.lookup("metric_1").from(lookup).as(`${lookupFields[0]} 1`), vl.lookup("metric_2").from(lookup).as(`${lookupFields[0]} 2`) ) .encode( vl.y().fieldN("metric_1").title(thisMetricName), vl.x().fieldN("metric_2").title(thisMetricName), vl.color().fieldQ("Correlation").scale({ scheme: colorScheme }), vl.tooltip([ `${lookupFields[0]} 1`, `${lookupFields[0]} 2`, { field: "Correlation", format: "r" } ]), vl .opacity() .if(correlationsRadioButtons.isolation, vl.value(1)) .value(0.09) ) .render(); }36:[{"id":"3ecced0eb8cdeb388a6a2b076a5f758a0da91d00ce4eca327e84a680d3690ac8b8ee3c5fe0e4b70d548f6debf522e71d9db57bfa7b0696f66374d159fa465f25","href":"https://static.observableusercontent.com/files/3ecced0eb8cdeb388a6a2b076a5f758a0da91d00ce4eca327e84a680d3690ac8b8ee3c5fe0e4b70d548f6debf522e71d9db57bfa7b0696f66374d159fa465f25","name":"DJIA2020.csv","type":"text/csv","lastModified":1614620830521.524,"size":51508}] 30:{"title":"Dow Jones Industrial Average Correlation Analysis with Arquero","cells":[{"id":0,"value":"md`# Dow Jones Industrial Average Correlation Analysis with Arquero`","pinned":false,"hidden":false,"mode":"ojs"},{"id":773,"value":"correlationHeatMap = html`${viewof correlationsRadioButtons}${viewof colorScheme}${heatMap}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":246,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":465,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":378,"value":"md`## Data\nFor this notebook, *data* is an array of objects created with [*GoogleFinance*](https://support.google.com/docs/answer/3093281?hl=en) that contains DJIA stock symbols and their respective closing prices; for each trading day in year 2020:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"data = {\n const data = await FileAttachment(\"DJIA2020.csv\").csv({ typed: true });\n data._fromNotebook = true;\n return data;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":610,"value":"md `*stockCompany* translates stock symbols to their respective company names.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":605,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":255,"value":"md`## Data Profile\n[*Datalib*](https://github.com/vega/datalib)'s [*summary*](https://github.com/vega/datalib/wiki/Statistics#dl_summary) function is used to create a data profile of the data to identify all numeric columns:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":10,"value":"// https://github.com/vega/datalib/wiki/Statistics#dl_summary\ndataProfile = dl.summary(data)","pinned":true,"hidden":false,"mode":"ojs"},{"id":264,"value":"md `## Correlation Pairs \n*metrics* is an array that filters out just the numeric columns in the data profile. These will be the variables analyzed for the correlations.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":19,"value":"metrics = dataProfile\n .filter(({ type, field }) => type == \"number\")\n .map(({ field }) => field)","pinned":true,"hidden":false,"mode":"ojs"},{"id":29,"value":"md `The metrics are transformed into correlation pairs for the analysis; the [*2-combination*](https://en.wikipedia.org/wiki/Combination) of the stock symbols:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1215,"value":"correlationPairs = {\n function combination2(set) {\n const pair = [];\n for (let i = 0; i < set.length; i++) {\n for (let j = i + 1; j < set.length; j++) {\n pair.push([set[i], set[j]]);\n }\n }\n return pair;\n } // function\n\n return combination2(metrics).map(([metric_1, metric_2]) => ({\n metric_1,\n metric_2\n }));\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":285,"value":"md `## Programmatically Building the Arquero Rollups\nWith the correlation pairings determined, the *rollupObj* is dynamically constructed. This object will be passed to the [*rollup*](https://observablehq.com/@uwdata/an-illustrated-guide-to-arquero-verbs#rollup) method to calculate the correlations for each metric pair. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":979,"value":"pairSeparator = ` : `;","pinned":false,"hidden":false,"mode":"ojs"},{"id":77,"value":"rollupObj = {\n const rollupObj = {};\n\n for (const { metric_1, metric_2 } of correlationPairs) {\n // https://uwdata.github.io/arquero/api/expressions\n // https://observablehq.com/@mariodelgadosr/arquero-correlation-speed-test\n rollupObj[\n `${metric_1}${pairSeparator}${metric_2}`\n ] = `d => op.corr(d[\"${metric_1}\"], d[\"${metric_2}\"])`;\n } // for\n\n return rollupObj;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":321,"value":"md`## Correlation Analysis\n*correlationAnalysis* is a reference to the original dataset:\n\n* Loaded into an Arquero table;\n* With correlations calculated for each of the metric pairs using the programmatically generated *rollupObj*. \n\nThe results are then used as the input to the [Vega-lite](https://observablehq.com/@uwdata/introduction-to-vega-lite) visualization. \n\nThe analysis reveals:\n\n${maxMin.view()}\n\n* ${analysisTextMax};\n* ${analysisTextMin}.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"import { aq, op } from '@uwdata/arquero'","pinned":false,"hidden":false,"mode":"ojs"},{"id":983,"value":"correlationAnalysis = aq\n .from(data)\n .rollup(rollupObj)\n .fold(aq.all(), { as: [\"Correlation Pair\", \"Correlation\"] })\n // https://uwdata.github.io/arquero/api/expressions#limitations\n .derive(\n {\n metric_1: `d => op.split(d[\"Correlation Pair\"],\"${pairSeparator}\")[0]`,\n metric_2: `d => op.split(d[\"Correlation Pair\"],\"${pairSeparator}\")[1]`\n },\n { before: \"Correlation Pair\" }\n )\n .orderby(aq.desc(\"Correlation\"))","pinned":true,"hidden":false,"mode":"ojs"},{"id":305,"value":"correlationAnalysis.view(correlationAnalysis.numRows()); ","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"maxMin = correlationAnalysis\n .derive({\n maxPair: op.first_value(\"Correlation Pair\"),\n minPair: op.last_value(\"Correlation Pair\")\n })\n .rollup({\n max: (d) => op.max(d.Correlation),\n min: (d) => op.min(d.Correlation),\n maxPair: op.max(\"maxPair\"),\n minPair: op.min(\"minPair\")\n })","pinned":true,"hidden":false,"mode":"ojs"},{"id":789,"value":"md`## Texts, Viewofs, HeatMap and Imports`","pinned":false,"hidden":false,"mode":"ojs"},{"id":705,"value":"analysisTextMax = \n`Max correlation of ${maxMin.get(\"max\",0).toFixed(6)} for ${maxMin.get(\"maxPair\",0)}`;","pinned":true,"hidden":false,"mode":"ojs"},{"id":708,"value":"analysisTextMin = \n`Min correlation of ${maxMin.get(\"min\",0).toFixed(6)} for ${maxMin.get(\"minPair\",0)}`;","pinned":true,"hidden":false,"mode":"ojs"},{"id":906,"value":"viewof correlationsRadioButtons = {\n const minMaxPairs = maxMin\n .get(\"maxPair\", 0)\n .split(pairSeparator)\n .concat(maxMin.get(\"minPair\", 0).split(pairSeparator));\n const maxMinFilter = `(datum.metric_1 == '${minMaxPairs[0]}' && datum.metric_2 == '${minMaxPairs[1]}') || \n (datum.metric_1 == '${minMaxPairs[2]}' && datum.metric_2 == '${minMaxPairs[3]}')`;\n const isolation = [\n \"true\",\n maxMinFilter,\n \"datum.Correlation > 0.5\",\n \"datum.Correlation < - 0.5\"\n ];\n const subtitle = [null, `${analysisTextMax}; ${analysisTextMin}`, null, null];\n const correlations = [\"All\", \"Max/Min\", \"> 0.5\", \"< -0.5\"];\n\n const corrMap = new Map(\n correlations.map((corr, i) => [\n corr,\n { isolation: isolation[i], subtitle: subtitle[i] }\n ])\n );\n\n return Inputs.radio(corrMap, {\n label: \"Correlations\",\n value: corrMap.get(\"All\")\n });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":585,"value":"// https://vega.github.io/vega/docs/schemes/#diverging\nviewof colorScheme = {\n const schemes = new Map([\n [\"Blue-Orange\", \"blueorange\"],\n [\"Brown-Blue-Green\", \"brownbluegreen\"],\n [\"Puple-Green\", \"purplegreen\"],\n [\"Pink-Yellow-Green\", \"pinkyellowgreen\"],\n [\"Purple-Orange\", \"purpleorange\"],\n [\"Red-Blue\", \"redblue\"],\n [\"Red-Grey\", \"redgrey\"],\n [\"Red-Yellow-Blue\", \"redyellowblue\"],\n [\"Red-Yellow-Green\", \"redyellowgreen\"],\n [\"Spectral\", \"spectral\"]\n ]);\n\n return Inputs.select(schemes, { label: \"Color Scheme\", value: \"spectral\" });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1308,"value":"// Name of the metric in the correlation analysis can be overriden when importing \nmetricName = undefined","pinned":true,"hidden":false,"mode":"ojs"},{"id":1314,"value":"// The heatmap's title can be overriden when importing\ntitle = undefined","pinned":true,"hidden":false,"mode":"ojs"},{"id":1315,"value":"// A metric's lookup table can be overriden. See stockCompany for an example of a lookupTable key/value format\nlookupTable = undefined","pinned":true,"hidden":false,"mode":"ojs"},{"id":496,"value":"$35","pinned":true,"hidden":false,"mode":"ojs"},{"id":492,"value":"import {vl} from \"@vega/vega-lite-api\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":1199,"value":"// https://github.com/vega/datalib\ndl = require(\"datalib\")","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}