31:T5ed,md`## Overview This notebook visualizes USDA nutrient data (obtained from the [Nutrient Explorer](http://bl.ocks.org/syntagmatic/raw/3150059/)) using the dimensionality reduction technique [UMAP](https://umap-learn.readthedocs.io/en/latest/) (top/left) and the interactive heatmap tool [Clustergrammer-GL](https://github.com/ismms-himc/clustergrammer-gl) (bottom/right). This notebook is best viewed in landscape mode and in Fullscreen mode (see Observable three dots menu). We see there are three broad types of food (dots in the UMAP and columns in the heatmap): 1) high in water content and protein (left cluster of mostly meats), 2) high in water content and low in protein (veggies, soups, fruit), and 3) low in water content and high in calories (baked goods, sweets, fast foods, etc.). These three clusters can also be seen in the UMAP as the left, middle-top, and right clusters of foods. We can of course find more fine-grained clusters of food groups as well by adjusting the column dendrogram slider in the heatmap or by zooming into the UMAP. We also see that nutrients (dimensions/rows) also form clusters (e.g. related fat measures) and show anti-correlated behavior (e.g. water and calories are anti-correlated). Clicking on the rows/dimensions of the heatmap overlays the dimension data on the UMAP. Clicking on the column dendrogram highlights the selected foods on the UMAP and shows how we observe similar clustering of foods using two independent techniques - UMAP and hierarchcial clustering. `32:T713,md` ## Crepes and Cakes

Photo: [Source](https://us.kiwilimon.com/recipe/desserts/oreo-crepes-cake)

In a way, high-dimensional data visualiation is a lot like a crepe cake. A UMAP can be thought of as viewing a single crepe head-on, which shows a detailed two-dimensional representation of our data, but limits us to showing one or a few attributes at a time (e.g. we can color the data points using the values of a dimension). Whereas a heatmap can be thought of as viewing many layers of a crepe side-on (e.g. a crepe-cake), which shows a more restricted one-dimensional representation of our data points, but allows us to layer many attributes (e.g. we can simultaneously view many categorical and data dimensions). By interactively linking our UMAP and heatmap we can get the best of both worlds. Clicking on a row (e.g. nutrient dimension) can be thought of as pulling out a single crepe from the cake and viewing it head-on and colored by the specific layer/nutrient dimension. Clicking on a cluster of foods (e.g. the column dendrogram) shows us where our cluster appears in our UMAP dimensionality reduced space and helps us identify robust clusters. We can also of course view UMAPs of the nutrient dimensions (not shown) which is equivalent to taking vertical slices from the crepe-cake. The heatmap shows us that nutrients themselves form clusters and some are anti-correlated (e.g. water and calories). ## Data Preprocessing Preprocessing steps can be found at https://github.com/cornhundred/USDA_Nutrients_Viz. The UMAP visualization was built using [deck.gl](https://deck.gl/). `33:T6d7,viewof cgm = { // Clustergrammer-GL uses query selectors to re-find the container, so if we // might want multiple instances of Clustergrammer-GL on the page, they’ll // need unique identifiers! (It’d be better if Clustergrammer-GL used // element.querySelector instead of document.querySelector and didn’t depend // on these identifiers.) const containerId = DOM.uid("container"); const container = html`

`; // Clustergrammer-GL depends on the container having been inserted into the // document before it’s initialized. We can do this by yielding! However, // since we’re defining this cell as a view, we don’t want the view’s value // to resolve until CGM has initialized. Hence we define the view’s value as // a promise that resolves immediately after. We could simplify this if // Clustergrammer-GL allowed you to pass in a detached container. let resolve; container.value = new Promise(r => resolve = r); yield container; // Initialize CGM. // (We might want to use Observable’s reactive width here?) const cgm = CGM({ network, viz_width: 900, viz_height: 900, container }); // Redefine the order parameters as getters and setters, so that we can intercept // when these change and emit an event. That will allow other cells to listen for // these changes are react accordingly. Object.defineProperties(cgm.params.order.new, { row: reactiveValue(container, "row", cgm.params.order.new.row), col: reactiveValue(container, "col", cgm.params.order.new.col) }); // Resolve the view’s value so that downstream cells can run. resolve(cgm); d3.select('.row_search_container').remove() }34:T4c5,umap_data = umap_data_ini.map( x => { var inst_long = umap_x_scale(parseFloat(x['x'])) var inst_lat = umap_y_scale(parseFloat(x['y'])) var inst_food_type = food_type[x['']].split(': ')[1] var rgb = d3.color(cat_colors[inst_food_type]) var inst_color if (selection === 'category'){ // color by category var inst_alpha if (selection_type == 'col-dendro'){ if (dendro_selections.includes(x[''])){ inst_alpha = 255 } else { inst_alpha = 25 } } else { inst_alpha = 255 } inst_color = [rgb.r, rgb.g, rgb.b, inst_alpha] } else { // color by dimension var inst_index = network.col_node_names.indexOf(x['']) var inst_data = dimension_data[inst_index] var inst_opacity = opacity_scale(Math.abs(inst_data)) if (inst_data > 0){ inst_color = [255, 0, 0, inst_opacity] } else { inst_color = [0, 0, 255, inst_opacity] } } var inst_obj = ({ name: x[''], position: [inst_long, inst_lat], food_type: inst_food_type, color: inst_color }) return inst_obj } )35:T416,scatter_layer_rename = { const scatterLayer = new deck.ScatterplotLayer({ data: umap_data, getPosition: d => d.position, getFillColor: d => d.color, getRadius: 300, opacity: 0.75, pickable: true, autoHighlight: true, highlightColor: d => [50, 50, 50, 200], // stroked: true, // lineWidthMinPixels: 0.05, onHover: function(info, event){ // if ('object' in info){ // mutable inst_neighborhood = info.object.Neighborhood // } }, onClick: function(info, event){ // if ('object' in info){ // if (info.object.name !== selected_station.name){ // // select stations to view paths // mutable path_types = 'all' // mutable selected_station = info.object // } else { // // turn off paths // mutable selected_station = {} // } // } } }) deckgl.setProps({layers: [ scatterLayer]}); return scatterLayer }36:[{"id":"29802281091d30af8304c4f1906d84f7b6c90290d9f1668fc5f3e5939d2d7429c5dce1e85d34ac40c9366d311369eadb141eecbdd9f00e4430a55441511d6de2","href":"https://static.observableusercontent.com/files/29802281091d30af8304c4f1906d84f7b6c90290d9f1668fc5f3e5939d2d7429c5dce1e85d34ac40c9366d311369eadb141eecbdd9f00e4430a55441511d6de2","name":"usda_foods_umap.csv","type":"text/csv","lastModified":1608745795220.493,"size":588871},{"id":"f127b5d83ea81e0fff576b86e6136d0db2c68c5af8c125c4d319282a4c20a902721f0eae94b0c333e230d2e706f1be59e0b38a4a4fa8687183088c25cb29814c","href":"https://static.observableusercontent.com/files/f127b5d83ea81e0fff576b86e6136d0db2c68c5af8c125c4d319282a4c20a902721f0eae94b0c333e230d2e706f1be59e0b38a4a4fa8687183088c25cb29814c","name":"Viz_USDA_Nutrients@1.json","type":"application/json","lastModified":1606800577438.313,"size":4040973},{"id":"528cd8d183651c3cd33f113b0f22843a943d2a6a2782ba27650081ba1ec6c3c6a1bac2241163346846da6f4f00cb93276be9c94ee40dbfea5cbc76102077a276","href":"https://static.observableusercontent.com/files/528cd8d183651c3cd33f113b0f22843a943d2a6a2782ba27650081ba1ec6c3c6a1bac2241163346846da6f4f00cb93276be9c94ee40dbfea5cbc76102077a276","name":"clustergrammer-gl.0.19.0.packd.js","type":"application/javascript","lastModified":1606800178439.85,"size":1017517},{"id":"b873c5954a40444b4c5e34bd7f2478c9aa20c8e438be2b9f5687d2624a00ab29e8ef0ea5d824b0d6f69a42513d1cdcfcb1a1ba348557a81fc18c1b4ceb9a46d8","href":"https://static.observableusercontent.com/files/b873c5954a40444b4c5e34bd7f2478c9aa20c8e438be2b9f5687d2624a00ab29e8ef0ea5d824b0d6f69a42513d1cdcfcb1a1ba348557a81fc18c1b4ceb9a46d8","name":"oreo_crepe_cake.jpg","type":"image/jpeg","lastModified":1608746899592.493,"size":193384}] 2f:{"title":"USDA Nutrients UMAP-Heatmap","cells":[{"id":0,"value":"md`\n# USDA Nutrients UMAP-Heatmap\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":298,"value":"viewof orientation = select({\n description: \"Select Orientation - switch to fullscreen for Landscape\",\n options: [\"Portrait\", \"Landscape\"],\n value: \"Portrait\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":297,"value":"clustergrammer = { \n\n if (orientation == 'Portrait' && is_fullscreen === false){\n \n var inst_html = html`\n

\n ${container}

${viewof cgm}

\n \n `\n } else {\n\n \n var inst_html = html`

${container}

${viewof cgm}

`\n }\n \n return inst_html\n\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":375,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":236,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"md`\n## Suggested listening: Mm..Food?\n\n[![](https://media.pitchfork.com/photos/5929c44fc0084474cd0c3670/1:1/w_320/4446b004.jpg)](https://pitchfork.com/reviews/albums/5255-mmfood/)\n\nThis classic MF DOOM (RIP) album makes great companion listening while exploring high-dimensional food and nutrient data.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":280,"value":"md`## Code, Data, and Imports`","pinned":false,"hidden":false,"mode":"ojs"},{"id":295,"value":"selection_type = Generators.observe(notify => {\n\n const mousemove = () => notify(cgm.params.tooltip.tooltip_type);\n \n viewof cgm.addEventListener(\"click\", mousemove);\n notify(cgm.params.tooltip.tooltip_type);\n \n return () => viewof cgm.removeEventListener(\"mousemove\", mousemove);\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":294,"value":"dendro_selections = Generators.observe(notify => {\n\n const mousemove = () => {\n \n if (cgm.params.tooltip.tooltip_type.includes('dendro')){\n mutable selection = 'category'\n }\n\n notify(cgm.params.dendro.selected_clust_names)\n }\n \n viewof cgm.addEventListener(\"click\", mousemove);\n notify(cgm.params.dendro.selected_clust_names);\n\n return () => viewof cgm.removeEventListener(\"mousemove\", mousemove);\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":292,"value":"mutable selection = 'category'","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"selected_row_index = network.row_node_names.indexOf(selected_row)","pinned":false,"hidden":false,"mode":"ojs"},{"id":290,"value":"dimension_data = network.mat[selected_row_index]","pinned":false,"hidden":false,"mode":"ojs"},{"id":289,"value":"container = html `

`","pinned":true,"hidden":false,"mode":"ojs"},{"id":288,"value":"mutable is_fullscreen = false","pinned":true,"hidden":false,"mode":"ojs"},{"id":26,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":286,"value":"function reactiveValue(container, name, value) {\n return {\n get() {\n return value;\n },\n set(v) {\n value = v;\n container.dispatchEvent(new CustomEvent(\"input\", {detail: {name, value}}));\n return value;\n }\n };\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":285,"value":"mutable color_type = 'category'","pinned":true,"hidden":false,"mode":"ojs"},{"id":283,"value":"mutable selected_row = Generators.observe(notify => {\n const mouse_click = () => {\n\n if (cgm.params.tooltip.tooltip_type === 'row-label'){\n var inst_name = cgm.params.int.mouseover.row.name\n if (selection != inst_name){\n mutable selection = inst_name\n } else {\n mutable selection = 'category'\n }\n }\n \n notify(cgm.params.int.mouseover.row.name)\n }\n \n viewof cgm.addEventListener(\"click\", mouse_click);\n \n notify(cgm.params.int.mouseover.row.name);\n\n return () => viewof cgm.removeEventListener(\"mousemove\", mouse_click);\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":279,"value":"umap_data_ini = d3.csvParse(await FileAttachment(\"usda_foods_umap.csv\").text())","pinned":true,"hidden":false,"mode":"ojs"},{"id":278,"value":"food_type = {\n var food_type = {}\n network.col_nodes.map(x => {\n food_type[x.name] = x['cat-0']\n })\n return food_type\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":277,"value":"cat_colors = {\n var cat_colors = {} \n var cat_colors_ini = network.cat_colors.col['cat-0']\n var ini_keys = Object.keys(cat_colors_ini)\n \n ini_keys.map(x => {\n cat_colors[x.split(': ')[1]] = cat_colors_ini[x]\n })\n return cat_colors\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":276,"value":"opacity_scale = d3.scaleLinear()\n .domain([0, 5])\n .range([0, 255]);","pinned":false,"hidden":false,"mode":"ojs"},{"id":271,"value":"$34","pinned":true,"hidden":false,"mode":"ojs"},{"id":270,"value":"d3 = require(\"d3@5\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":269,"value":"inst_lat = 40.5","pinned":true,"hidden":false,"mode":"ojs"},{"id":268,"value":"inst_lng = -73.5","pinned":true,"hidden":false,"mode":"ojs"},{"id":267,"value":"inst_zoom = 8.75","pinned":true,"hidden":false,"mode":"ojs"},{"id":266,"value":"inst_height = {\n var inst_height\n if (orientation === 'Portrait'){ \n inst_height = 850\n } else {\n inst_height = 850\n }\n return inst_height\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":265,"value":"umap_data[0]","pinned":true,"hidden":false,"mode":"ojs"},{"id":264,"value":"umap_x_min = Math.min.apply(Math, umap_data_ini.map(function(o) { return parseFloat(o['x']) }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":263,"value":"umap_x_max = Math.max.apply(Math, umap_data_ini.map(function(o) { return parseFloat(o['x']) }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":262,"value":"umap_y_min = Math.min.apply(Math, umap_data_ini.map(function(o) { return parseFloat(o['y']) }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":261,"value":"umap_y_max = Math.max.apply(Math, umap_data_ini.map(function(o) { return parseFloat(o['y']) }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":260,"value":"umap_x_scale = d3.scaleLinear()\n .domain([umap_x_min, umap_x_max])\n .range([-74, -73]);","pinned":true,"hidden":false,"mode":"ojs"},{"id":259,"value":"umap_y_scale = d3.scaleLinear()\n .domain([umap_y_min, umap_y_max])\n .range([40, 41]);","pinned":true,"hidden":false,"mode":"ojs"},{"id":258,"value":"umap_data[0]","pinned":true,"hidden":false,"mode":"ojs"},{"id":257,"value":"$35","pinned":true,"hidden":false,"mode":"ojs"},{"id":256,"value":"deckgl = {\n \n // // reference for below https://observablehq.com/@tomvantilburg/deckgl-mapbox-and-3d-tiles\n // // This is an Observable hack: clear previously generated content\n container.innerHTML = '';\n \n return new deck.DeckGL({\n container,\n initialViewState: {\n longitude: inst_lng,\n latitude: inst_lat,\n zoom: inst_zoom,\n minZoom: 8,\n maxZoom: 15, \n },\n controller: true,\n\n getTooltip: ({object}) => {\n // return object && `${object.Neighborhood} \\n ${object['cross-x']} and ${object['cross-y']}\n return object && `${object.name}\\n Food Type: ${object.food_type}` },\n // Mapbox settings\n // set `map: false` to turn off Mapbox base map\n // map: mapboxgl,\n // This token is for demo-purpose only and rotated regularly. Get your token at https://www.mapbox.com\n // mapboxApiAccessToken: mapbox_token,\n mapStyle: 'mapbox://styles/mapbox/light-v9',\n });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":255,"value":"// deck = require.alias({\n// // optional dependencies\n// h3: {},\n// s2Geometry: {}\n// })('deck.gl@8.3.7/dist.min.js')\n\ndeck = require.alias({\n // optional dependencies\n h3: {},\n s2Geometry: {}\n})('deck.gl@8.4.6/dist.min.js')","pinned":true,"hidden":false,"mode":"ojs"},{"id":137,"value":"// network = fetch(\"https://raw.githubusercontent.com/MaayanLab/clustergrammer/master/json/mult_view.json\")\n// .then(response => response.json())\nnetwork = FileAttachment(\"Viz_USDA_Nutrients@1.json\").json()","pinned":false,"hidden":false,"mode":"ojs"},{"id":212,"value":"// cgm = require(\"https://bundle.run/clustergrammer-gl@0.11.9\")\n// cgm = require(await FileAttachment(\"clustergrammer-gl.0.11.10.packd.js\").url())\nCGM = require(await FileAttachment(\"clustergrammer-gl.0.19.0.packd.js\").url())","pinned":false,"hidden":false,"mode":"ojs"},{"id":253,"value":"import {slider, select} from '@jashkenas/inputs'","pinned":true,"hidden":false,"mode":"ojs"},{"id":352,"value":"// set table style to full width \nhtml`\n