2f:T49a, ## Labs similarity & Keywords co-occurence We quantify relationships using two metrics: co-occurrence and similarity. Co-occurrence, derived from a matrix, represents __how often pairs of keywords appear in the same lab's list__. Higher weights signify frequent co-occurrences. Cosine similarity, calculated by comparing labs' keyword lists, indicates the __degree of connection between labs__. More shared, high-weight keywords result in greater similarity. These measures elucidate the complex ties between labs and keywords. ## Interactives graphs Both graphs visualize the connections between labs and keywords. The keyword graph uses a __co-occurrence matrix__ to highlight links between keywords that frequently appear together in a lab's keyword list. The lab graph, on the other hand, illustrates connections between labs according to their __cosine similarity__. When you hover over a lab, associated high-weight keywords are highlighted on the keyword graph. Similarly, hovering over a keyword illuminates relevant labs on the lab graph. These graphs offer an interactive, intuitive means to understand the __relationships and patterns between labs and keywords__.30:T421,{ const labs = createGraphLabs(width / 2, height); const keywords = createGraphKeywords(width * 0.75, height); // create a container let container = document.createElement("div"); container.style.backgroundColor = "white"; // container.style.padding = "30px"; container.style.display = "flex"; container.style.justifyContent = "space-between"; container.style.height = `${height}px`; // append SVGs to the container container.appendChild(labs); container.appendChild(keywords); labs.addEventListener("input", (event) => { const selectedNode = event.target.value; if (selectedNode) { keywords.filterNodes(selectedNode.topKeywords); } }); labs.addEventListener("mouseleave", () => { keywords.filterNodes([]); }); keywords.addEventListener("input", (event) => { const words = event.target.value; if (words && words.length > 0) { labs.filterNodes(words); } }); keywords.addEventListener("mouseleave", () => { labs.filterNodes([]); }); return Object.assign(container); }31:T424,// Copyright 2021 Observable, Inc. // Released under the ISC license. // https://observablehq.com/@d3/sunburst sunburst_interactive = { const data = createTree(commonKeywordsTree, "knowledge", 4); const graph = createGraphLabs(width / 2, height); const sunburst = createSunburst(data, { sWidth: width / 2, height: width / 2 }); // create a container let container = document.createElement("div"); container.style.backgroundColor = "white"; // container.style.padding = "30px"; container.style.display = "flex"; container.style.justifyContent = "space-between"; container.style.height = `${height}px`; // append SVGs to the container container.appendChild(sunburst); container.appendChild(graph); graph.addEventListener("input", (event) => { const selectedNode = event.target.value; if (selectedNode) { const sunburst_weights = labs_trees.get(selectedNode.id).sunburst_weights; sunburst.filterTree(sunburst_weights, selectedNode.id); } }); return Object.assign(container, { ...sunburst }); }32:T4d1,We use two metrics to quantify the relationships between different labs: Weight and Cosine Similarity. - **Weight**: The weight of a connection between two labs is calculated as the sum of the average weights of the common keywords shared by the labs. This value measures the overall significance of shared keywords, providing an indication of how strongly two labs are connected based on shared topics of interest. A higher weight implies a greater degree of shared interest, as evidenced by keyword overlap. - **Cosine Similarity**: As a complementary measure, we use cosine similarity to assess the similarity in the distribution of keywords between two labs. Each lab is represented as a vector in a multi-dimensional space, where each dimension corresponds to a keyword and its magnitude represents the keyword's significance. Cosine similarity, hence, captures how similar the keyword distributions are between two labs, regardless of their magnitude. This provides a broader perspective on the shared research interests between labs, transcending mere keyword overlap to consider the overall distribution. The graph below show correlation beetween these two values, including the similarity threshold selected (default 0.1)33:T1634,function createGraphKeywords(width, height) { let simulationLinks = JSON.parse(JSON.stringify(wordLinks)); let simulationNodes = topOverallKeywords.map(([name, value]) => ({ id: name, size: value, connectedKeywords: Object.values(keywordLinks.get(name) ?? []).map( ({ source, target }) => (source == name ? target : source) ) })); let nodeSizeScale = d3 .scaleSqrt() .domain(d3.extent(simulationNodes, (d) => d.size)) // Use the range of your link weights .range([4, 50]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let fontSizeScale = d3 .scaleSqrt() .domain(d3.extent(simulationNodes, (d) => d.size)) // Use the range of your link weights .range([12, 50]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let linkStrengthScale = d3 .scaleLinear() .domain(d3.extent(simulationLinks, (d) => d.weight)) // Use the range of your link weights .range([0.1, 6]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let linkDistanceScale = d3 .scaleLinear() .domain(d3.extent(simulationLinks, (d) => d.weight)) // Use the range of your link weights .range([200, 100]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let simulation = d3 .forceSimulation(simulationNodes) .force( "link", d3 .forceLink(simulationLinks) .id((d) => d.id) .strength((d) => linkStrengthScale(d.weight)) // Apply the link strength scale here // .distance((d) => linkDistanceScale(d.weight)) ) .force("charge", d3.forceManyBody().strength(-10000)) .force( "x", d3 .forceX() .x(width / 2) .strength(1.5) ) .force( "y", d3 .forceY() .y(height / 2) .strength(1.5) ) .force( "collide", d3.forceCollide().radius((d) => nodeSizeScale(d.size) * 3) ); // Create an svg to hold the visualization and apply the zoom behavior to it let svg = d3.create("svg").attr("viewBox", [0, 0, width, height]), g = svg.append("g"); const input = svg.node(); // Create lines for the simulationLinks let link = g .append("g") .attr("stroke", "#999") .attr("stroke-opacity", 0.6) .attr("stroke-width", 0.4) .selectAll("line") .data(simulationLinks) .join("line"); // Create circles for the simulationNodes let node = g .append("g") .selectAll("g") .data(simulationNodes) .enter() .append("g") .style("cursor", "move"); node .append("circle") .attr("r", (d) => nodeSizeScale(d.size) * 3) .attr("fill", "transparent"); node .append("circle") .attr("r", (d) => nodeSizeScale(d.size)) .attr("fill", (d) => conceptColorScale(conceptAncestor.get(d.id))) .attr("fill-opacity", 0.8) .attr("stroke", "#fff") .attr("stroke-width", 1.5); node .append("text") .text((d) => d.id) .attr("x", 0) .attr("y", 3) .attr("font-size", (d) => `${fontSizeScale(d.size)}px`); // Add drag behavior to the simulationNodes node.call(drag(simulation)); node.on("mouseover", (event, d) => { node.style("opacity", (o) => d.id == o.id || d.connectedKeywords.includes(o.id) || o.connectedKeywords.includes(d.id) ? 1 : 0.2 ); link.style("opacity", (l) => l.source.id == d.id || l.target.id == d.id ? 1 : 0.1 ); input.value = d.connectedKeywords.concat([d.id]); input.dispatchEvent(new Event("input", { bubbles: true })); }); svg.on("click", (event, d) => { node.style("opacity", 1); link.style("opacity", 1); }); svg.on("mouseout", (event, d) => { node.style("opacity", 1); link.style("opacity", 1); input.value = []; input.dispatchEvent(new Event("input", { bubbles: true })); }); // Update the positions of the nodes and links at each step of the simulation simulation.on("tick", () => { link .attr("x1", (d) => d.source.x) .attr("y1", (d) => d.source.y) .attr("x2", (d) => d.target.x) .attr("y2", (d) => d.target.y); node.attr("transform", (d) => `translate(${d.x},${d.y})`); }); // Define the drag behavior function drag(simulation) { function dragstarted(event, d) { if (!event.active) simulation.alphaTarget(0.01).restart(); d.fx = d.x; d.fy = d.y; } function dragged(event, d) { d.fx = event.x; d.fy = event.y; } function dragended(event, d) { if (!event.active) simulation.alphaTarget(0); d.fx = null; d.fy = null; } return d3 .drag() .on("start", dragstarted) .on("drag", dragged) .on("end", dragended); } function zoomed({ transform }) { g.attr("transform", transform); } function filterNodes(array) { if (!array || array.length == 0) { node.style("opacity", 1); link.style("opacity", 1); } else { node.style("opacity", (d) => (array.includes(d.id) ? 1 : 0.1)); link.style("opacity", (l) => array.includes(l.source.id) && array.includes(l.target.id) ? 1 : 0.1 ); } } svg.call( d3 .zoom() .extent([ [0, 0], [width, height] ]) .scaleExtent([0.2, 3]) .on("zoom", zoomed) ); return Object.assign(input, { filterNodes }); }34:T148a,// Copyright 2021 Observable, Inc. // Released under the ISC license. // https://observablehq.com/@d3/sunburst createSunburst = ( tree, { id = Array.isArray(tree) ? (d) => d.id : null, // if tabular data, given a d in data, returns a unique identifier (string) parentId = Array.isArray(tree) ? (d) => d.parentId : null, // if tabular data, given a node d, returns its parent’s identifier value = (d) => d.value, // given a node d, returns a quantitative value (for area encoding; null for count) sort = (a, b) => d3.descending(a.value, b.value), // how to sort nodes prior to layout label = (d) => d.name, // given a node d, returns the name to display on the rectangle title = (d, n) => `${n .ancestors() .reverse() .map((d) => d.data.name) .join(".")}\n${n.value.toLocaleString("en")}`, // hover text, // given a node d, returns its hover text linkTarget = "_blank", // the target attribute for links (if any) link, path, children, sWidth = width / 2, height = width / 2, // outer height, in pixels margin = 1, // shorthand for margins marginTop = margin, // top margin, in pixels marginRight = margin, // right margin, in pixels marginBottom = margin, // bottom margin, in pixels marginLeft = margin, // left margin, in pixels padding = 1, // separation between arcs radius = Math.min( sWidth - marginLeft - marginRight, height - marginTop - marginBottom ) / 2, // outer radius fill = "#ccc", // fill for arcs (if no color encoding) fillOpacity = 0.6 // fill opacity for arcs } = {} ) => { const data = tree; // If id and parentId options are specified, or the path option, use d3.stratify // to convert tabular data to a hierarchy; otherwise we assume that the data is // specified as an object {children} with nested objects (a.k.a. the “flare.json” // format), and use d3.hierarchy. const root = path != null ? d3.stratify().path(path)(data) : id != null || parentId != null ? d3.stratify().id(id).parentId(parentId)(data) : d3.hierarchy(data, children); // Compute the values of internal nodes by aggregating from the leaves. value == null ? root.count() : root.sum((d) => Math.max(0, value(d))); // Sort the leaves (typically by descending value for a pleasing layout). if (sort != null) root.sort(sort); // Compute the partition layout. Note polar coordinates: x is angle and y is radius. d3.partition().size([2 * Math.PI, radius])(root); // Construct an arc generator. const arc = d3 .arc() .startAngle((d) => d.x0) .endAngle((d) => d.x1) .padAngle((d) => Math.min((d.x1 - d.x0) / 2, (2 * padding) / radius)) .padRadius(radius / 2) .innerRadius((d) => d.y0) .outerRadius((d) => d.y1 - padding); const svg = d3 .create("svg") .attr("viewBox", [ marginRight - marginLeft - sWidth / 2, marginBottom - marginTop - height / 2, sWidth, height ]) .attr("width", sWidth) .attr("height", height) .attr("style", "max-width: 100%; height: auto; height: intrinsic;") .attr("font-family", "sans-serif") .attr("font-size", 10) .attr("text-anchor", "middle"); const cell = svg.selectAll("a").data(root.descendants()).join("g"); const arcs = cell .append("path") .attr("d", arc) .attr( "fill", color ? (d) => conceptColorScale(d.ancestors().reverse()[1]?.data.name) : fill ) .attr("fill-opacity", fillOpacity); if (label != null) cell .filter((d) => ((d.y0 + d.y1) / 2) * (d.x1 - d.x0) > 10) .append("text") .attr("transform", (d) => { if (!d.depth) return; const x = (((d.x0 + d.x1) / 2) * 180) / Math.PI; const y = (d.y0 + d.y1) / 2; return `rotate(${x - 90}) translate(${y},0) rotate(${ x < 180 ? 0 : 180 })`; }) .attr("dy", "0.32em") .each(function (d) { const angleDiff = d.x1 - d.x0; const text = label(d.data, d); const words = text.split(/\s+/g); let displayText = ""; if (angleDiff < 0.1) { // choose an appropriate threshold for your needs displayText = text.length > 15 ? text.slice(0, 13) + "..." : text; // truncate if text is too long d3.select(this).text(displayText); } else { const tspan = d3.select(this).selectAll("tspan").data(words); tspan .join("tspan") .attr("x", 0) .attr("dy", (d, i) => (i ? "1.2em" : 0)) .text((d) => d); } }); if (title != null) cell.append("title").text((d) => title(d.data, d)); function filterTree(weights, name) { if (weights === null) { arcs.attr("fill-opacity", 0.6); } else { const getWeight = (d) => weights.get(label(d.data)); const getSaturate = (d) => { const weight = getWeight(d); return `saturate(${weight * 4 ?? 0})`; }; arcs .style("filter", getSaturate) .attr("fill-opacity", (d) => getWeight(d) && getWeight(d) > 0.01 ? 0.6 : 0.2 ); } } return Object.assign(svg.node(), { filterTree }); }35:T141b,function createGraphLabs(width, height) { let simulationLinks = JSON.parse(JSON.stringify(topLinks)); let simulationNodes = JSON.parse(JSON.stringify(nodes)); let nodeSizeScale = d3 .scaleSqrt() .domain(d3.extent(simulationNodes, (d) => d.size)) // Use the range of your link weights .range([10, 50]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let linkDistanceScale = d3 .scaleLinear() .domain(d3.extent(simulationLinks, (d) => d.weight)) // Use the range of your link weights .range([300, 70]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range let simulation = d3 .forceSimulation(simulationNodes) .force( "link", d3 .forceLink(simulationLinks) .id((d) => d.id) .strength((d) => d.weight ** 2) // Apply the link strength scale here // .distance((d) => linkDistanceScale(d.weight)) ) .force("charge", d3.forceManyBody().strength(-30000)) .force( "x", d3 .forceX() .x(width / 2) .strength(1.25) ) .force( "y", d3 .forceY() .y(height / 2) .strength(1.25) ) .force( "collide", d3.forceCollide().radius((d) => nodeSizeScale(d.size) * 3) ); // Create an svg to hold the visualization and apply the zoom behavior to it let svg = d3.create("svg").attr("viewBox", [0, 0, width, height]), g = svg.append("g"); const input = svg.node(); // Create lines for the simulationLinks let link = g .append("g") .attr("stroke", "#999") .attr("stroke-opacity", 0.6) .selectAll("line") .data(simulationLinks) .join("line"); // Create circles for the simulationNodes let node = g .append("g") .selectAll("g") .data(simulationNodes) .enter() .append("g") .style("cursor", "move"); node .append("circle") .attr("r", (d) => nodeSizeScale(d.size) * 3) .style("opacity", 0); let circles = node .append("circle") .attr("r", (d) => nodeSizeScale(d.size)) .attr("fill", "#69b3a2") .attr("stroke", "#fff") .attr("stroke-width", 1.5); node .append("text") .text((d) => d.id) .attr("x", 0) .attr("y", 3) .attr("font-size", "xxx-large"); node.on("mouseover", (event, d) => { input.value = d; input.dispatchEvent(new Event("input", { bubbles: true })); node.style("opacity", (o) => d.id == o.id || d.connectedLabs.includes(o.id) || o.connectedLabs.includes(d.id) ? 1 : 0.3 ); node .selectAll("circle") .attr("fill", (o) => (d.id == o.id ? "green" : "#69b3a2")); link.style("opacity", (l) => l.source.id == d.id || l.target.id == d.id ? 1 : 0.3 ); // viewof keywords_graph.value.filterNodes(d.topKeywords); }); const reset = () => { input.value = null; input.dispatchEvent(new Event("input", { bubbles: true })); node.style("opacity", 1); link.style("opacity", 1); node.selectAll("circle").attr("fill", "#69b3a2"); // viewof keywords_graph.value.filterNodes([]); }; svg.on("mouseout", reset).on("click", reset); // Add drag behavior to the simulationNodes node.call(drag(simulation)); // Update the positions of the nodes and links at each step of the simulation simulation.on("tick", () => { link .attr("x1", (d) => d.source.x) .attr("y1", (d) => d.source.y) .attr("x2", (d) => d.target.x) .attr("y2", (d) => d.target.y); node.attr("transform", (d) => `translate(${d.x},${d.y})`); }); // Define the drag behavior function drag(simulation) { function dragstarted(event, d) { if (!event.active) simulation.alphaTarget(0.6).restart(); d.fx = d.x; d.fy = d.y; } function dragged(event, d) { d.fx = event.x; d.fy = event.y; } function dragended(event, d) { if (!event.active) simulation.alphaTarget(0); d.fx = null; d.fy = null; } return d3 .drag() .on("start", dragstarted) .on("drag", dragged) .on("end", dragended); } function zoomed({ transform }) { g.attr("transform", transform); } function hasCommonElement(array1, array2) { let set = new Set(array1); return array2.some((element) => set.has(element)); } function filterNodes(array) { if (!array || array.length == 0) { node.style("opacity", 1); link.style("opacity", 1); } else { node.style("opacity", (d) => hasCommonElement(array, d.topKeywords) ? 1 : 0.1 ); link.style("opacity", (l) => hasCommonElement(array, l.source.topKeywords) && hasCommonElement(array, l.target.topKeywords) ? 1 : 0.1 ); } } svg.call( d3 .zoom() .extent([ [0, 0], [width, height] ]) .scaleExtent([0.5, 3]) .on("zoom", zoomed) ); return Object.assign(input, { filterNodes }); }36:T69a,function WordCloud( data, // should be in the form of { text , size} { marginTop = 0, // top margin, in pixels marginRight = 0, // right margin, in pixels marginBottom = 0, // bottom margin, in pixels marginLeft = 0, // left margin, in pixels width = 640, // outer width, in pixels height = 400, // outer height, in pixels maxWords = 250, // maximum number of words to extract from the text fontFamily = "sans-serif", // font family fontScale = 15, // base font size color = null, padding = 0, // amount of padding between the words (in pixels) rotate = 0, // a constant or function to rotate the words invalidation // when this promise resolves, stop the simulation } = {} ) { const svg = d3 .create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("font-family", fontFamily) .attr("text-anchor", "middle") .attr("style", "max-width: 100%; height: auto; height: intrinsic;"); const g = svg .append("g") .attr("transform", `translate(${marginLeft},${marginTop})`); const cloud = d3Cloud() .size([width - marginLeft - marginRight, height - marginTop - marginBottom]) .words(data) .padding(padding) .rotate(rotate) .font(fontFamily) .fontSize((d) => Math.sqrt(d.size) * fontScale) .on("word", ({ size, x, y, rotate, text }) => { g.append("text") .attr("font-size", size) .style("fill", color({ size, x, y, rotate, text })) .attr("transform", `translate(${x},${y}) rotate(${rotate})`) .text(text); }); cloud.start(); invalidation && invalidation.then(() => cloud.stop()); return svg.node(); }37:T451,wordclouds = { const sources = [ "keywords_epfl_graph", "keywords_infoscience", "keywords_epfl_graph_teaching", "keywords_google_scholar" ]; return new Map( sources.map((source) => { const words = d3 .rollups( labs.map((d) => d[source]).flat(), (v) => d3.sum(v, (d) => d[1]), (d) => d[0] ) .sort((a, b) => b[1] - a[1]) .slice(0, 150), weights = words.map((d) => d[1]); let wordSizeScale = d3 .scaleLinear() .domain(d3.extent(weights)) // Use the range of your link weights .range([0.1, 20]) // Adjust this range according to your preference .clamp(true); // Ensures that the scale never returns a value outside the output range return [ source, WordCloud( words.map((d) => ({ text: d[0], size: wordSizeScale(d[1]) })), { padding: 3, width, height: 600, color: (d) => conceptColorScale(conceptAncestor.get(d.text)) } ) ]; }) ); }38:Ta09,links = { // Creating an empty array to store network data let networkData = []; // Iterate over all labs for (let i = 0; i < labs.length; i++) { // Compare each lab with every other lab for (let j = i + 1; j < labs.length; j++) { const { name: sourceLab, extentWeight: sourceExtentWeight, normalizedKeywords: sourceKeywords } = labs[i]; const { name: targetLab, extentWeight: targetExtentWeight, normalizedKeywords: targetKeywords } = labs[j]; // Identify common keywords between the pair of labs const commonKeywords = d3 .rollups( [...sourceKeywords, ...targetKeywords], // If they have common keywords we calculate the mean of both lab weight for this common keywords (v) => v.length > 1 ? d3.mean(v, ([name, value]) => value) : undefined, ([name, value]) => name ) // Filter out any keywords not in common .filter(([name, value]) => value !== undefined) // Sort the common keywords by their mean weight .sort((a, b) => b[1] - a[1]); // // Determine the count of the top 10% of common keywords // const top10KeywordsIndex = Math.max(3, ~~commonKeywords.length); // // Slice out the top 10% of common keywords // const top10Keywords = commonKeywords.slice(0, top10KeywordsIndex); // If there are any common keywords, push a new entry into the network data if (commonKeywords.length > 0) { let sourceKeywordVector = {}, targetKeywordVector = {}; sourceKeywords.forEach(([name, value]) => { sourceKeywordVector[name] = value; }); targetKeywords.forEach(([name, value]) => { targetKeywordVector[name] = value; }); const similarity = cosineSimilarity( sourceKeywordVector, targetKeywordVector ); networkData.push({ source: sourceLab, target: targetLab, common: commonKeywords, keywords: commonKeywords.map(([name, value]) => name), // Calculate the total weight of the top 10% of common keywords // weight: ~~d3.sum(top10Keywords, ([name, value]) => value) similarity, weight: similarity, old_weight: d3.sum(commonKeywords, ([name, value]) => value) }); } } } // Return the network data return networkData .sort((a, b) => b.similarity - a.similarity) .map((d, index) => ({ ...d, index })); }39:T530,findHull = (nodesToFormat, padding) => { let options = { run: true, show: ["Unit hull", "Institut hull"], encodingFTE: "diskArea", hullPadding: 100 }; let nodes = nodesToFormat.map((d) => [d.x, d.y]); let centroid = nodes.reduce(reduceCentroid, [0, 0]); let distances = nodes.map((d) => dist(d, centroid)); let avgDist = d3.mean(distances); let maxDist = d3.max(distances); const circleHull = () => { let path = new Path2D(); path.moveTo( centroid[0] + maxDist + (0.5 * padding * options.hullPadding) / 100, centroid[1] ); path.arc( centroid[0], centroid[1], maxDist + (0.5 * padding * options.hullPadding) / 100, 0, 2 * Math.PI ); return { polygon: null, centroid, paddedPolygon: null, hull: path }; }; if (nodes.length < 5) return circleHull(); // let filteredNodes = nodes.filter( // (node) => dist(node, centroid) < avgDist * options.maxDistanceHull // ); let poly = d3.polygonHull(nodes), paddedPolygon = null, hull = null; if (poly.length < 3) return circleHull(); if (poly) { paddedPolygon = padHull(poly, (padding * options.hullPadding) / 100); hull = new Path2D(d3.line().curve(d3.curveBasisClosed)(paddedPolygon)); } return { polygon: poly, centroid, paddedPolygon, hull }; }3a:T4a7,matrix = { const t0 = performance.now(); // Start of performance time // create a 2D array with size equal to number of unique keywords let matrix = Array.from({ length: keywordList.length }, () => new Array(keywordList.length).fill(0) ); // create a map for quick keyword lookups let keywordToIndexMap = new Map(); for (let i = 0; i < keywordList.length; i++) { keywordToIndexMap.set(keywordList[i], i); } labs.forEach((lab) => { for (let i = 0; i < lab.normalizedKeywords.length; i++) { for (let j = i + 1; j < lab.normalizedKeywords.length; j++) { let keyword1Index = keywordToIndexMap.get(lab.normalizedKeywords[i][0]); let keyword2Index = keywordToIndexMap.get(lab.normalizedKeywords[j][0]); if (keyword1Index === undefined || keyword2Index === undefined) continue; const weight = lab.normalizedKeywords[i][1] * lab.normalizedKeywords[j][1]; // increment the co-occurrence count matrix[keyword1Index][keyword2Index] += weight; matrix[keyword2Index][keyword1Index] += weight; } } }); console.log(performance.now() - t0, "milliseconds"); return matrix; }3b:T421,wrangled_data = { let fileData = await FileAttachment("all_sources (1).json").json({ typed: true }); return Object.entries(fileData) .map(([name, data]) => ({ name, ...data })) .filter(({ name }) => (removeIIEGE ? name !== "iie-ge" : true)) .map((lab) => { lab.keywords_epfl_graph = Object.entries(lab.keywords_epfl_graph); lab.keywords_infoscience = Object.entries(lab.keywords_infoscience); lab.keywords_epfl_graph_teaching = Object.entries( lab.keywords_epfl_graph_teaching ); lab.keywords_google_scholar = Object.entries(lab.keywords_google_scholar); lab.size = Number(lab.size); lab.keywords = lab[keywords_source].sort((a, b) => b[1] - a[1]); lab.extentWeight = d3.extent(lab.keywords, (d) => d[1]); const scale = d3.scaleLinear().domain(lab.extentWeight).range([0, 1]); lab.normalizedKeywords = lab.keywords.map(([name, value]) => [ name, scale(value) ]); return lab; }) .filter((d) => d.normalizedKeywords.length > 0); }3c:[{"id":"84818c6fd4e0cc39e9d92f68180460e9a28cde2f0f7c664ceacc8a2b3fd2c5509241747ff18c9bdb9c73a555b538df41eacfda534907fd292649588ae21d7f9c","href":"https://static.observableusercontent.com/files/84818c6fd4e0cc39e9d92f68180460e9a28cde2f0f7c664ceacc8a2b3fd2c5509241747ff18c9bdb9c73a555b538df41eacfda534907fd292649588ae21d7f9c","name":"tree.json","type":"application/json","lastModified":1686003694117.092,"size":165351},{"id":"7a8d0c74e348670cf540aad84bcadce4ed70b6b276b19f91aee08d5c5be8e07b89b75bbe6fca69188e80054c6890795a5439cc47210e4318f0387ef827a9ae85","href":"https://static.observableusercontent.com/files/7a8d0c74e348670cf540aad84bcadce4ed70b6b276b19f91aee08d5c5be8e07b89b75bbe6fca69188e80054c6890795a5439cc47210e4318f0387ef827a9ae85","name":"concept_parent.json","type":"application/json","lastModified":1686003694117.092,"size":151083},{"id":"8f805716745d07c38c29ac0f0954aa83ff254988e36fc1bea849318d3bbf812d22166958fd2695ae0956fe9472e190af3581c82dc8f603ba0b7bf7cdeb75a5a3","href":"https://static.observableusercontent.com/files/8f805716745d07c38c29ac0f0954aa83ff254988e36fc1bea849318d3bbf812d22166958fd2695ae0956fe9472e190af3581c82dc8f603ba0b7bf7cdeb75a5a3","name":"all_sources (1).json","type":"application/json","lastModified":1686003694117.092,"size":964609}] 2d:{"title":"IIE Research Landscape Slides","cells":[{"id":4,"value":"# IIE Research Landscape Slides\n\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":2208,"value":"slide`\nSee [our repository](https://github.com/EPFL-ENAC/iie-research-landscape). Press __F__ to start\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2219,"value":"slide`\n__Data collection steps__\n\nWe start by _scrapping_ the various data sources to find keywords related to each lab. Keyword groups are retrieved or computed separately.\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2231,"value":"slide`\n\n\n__Research data__\n\nThe _score_ relating a keyword to a lab is a metric of __how much the lab has published on this concept__ (detected from publication's abstract).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2235,"value":"slide`\n\n\n\n__Teaching data__\n\nWe count the __number of occurrences__ of each keyword for every publication of current labs.\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2240,"value":"slide`\n\n\n\n\n__Google Scholar__\n\nDone by hand. Lab's heads are searched on Google Scholar and their fields of research are added with a __score of 1__.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1072,"value":"slide`\n\n\n\n__Source selection__\n\nSwitch beetween each of our sources to see how they differ.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1043,"value":"viewof keywords_source = Inputs.radio(\n [\n \"keywords_epfl_graph\",\n \"keywords_infoscience\",\n \"keywords_epfl_graph_teaching\",\n \"keywords_google_scholar\"\n ],\n {\n label: \"Select source data\",\n format: (v) => v.replace(\"keywords_\", \"\").replaceAll(\"_\", \" \"),\n value: \"keywords_epfl_graph\"\n }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1780,"value":"wordcloud = wordclouds.get(keywords_source)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1183,"value":"$2f","pinned":false,"hidden":false,"mode":"md"},{"id":2173,"value":"htl.html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":2076,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":414,"value":"viewof similarityThreshold = Inputs.range([0, 0.6], {\n label: \"Similarity threshold\",\n step: 0.01,\n value: 0.1\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":935,"value":"## Concept tree visualisation : sunburst","pinned":false,"hidden":false,"mode":"md"},{"id":1198,"value":"viewof fteWeighted = Inputs.toggle({\n label: \"FTE computed weight\",\n value: false\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1207,"value":"Here's the concept tree visualized as a sunburst. It shows the fields of research from EPFL IIE, sizes are calculated either by the \"wheight\" of keywords (explained at the top of notebook) or by wheight x FTE in the lab depending on the toggle above this cell. You can also see the concept tree as a JSON object right below.","pinned":false,"hidden":false,"mode":"md"},{"id":1304,"value":"Inputs.bind(\n Inputs.radio(\n [\n \"keywords_epfl_graph\",\n \"keywords_infoscience\",\n \"keywords_epfl_graph_teaching\",\n \"keywords_google_scholar\"\n ],\n {\n label: \"Select source data\",\n format: (v) => v.replace(\"keywords_\", \"\").replaceAll(\"_\", \" \"),\n value: \"keywords_epfl_graph\"\n }\n ),\n viewof keywords_source\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1204,"value":"hierarchy_tree_small","pinned":false,"hidden":false,"mode":"ojs"},{"id":1924,"value":"fullscreen = htl.html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":1468,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":1041,"value":"### Tidy tree","pinned":false,"hidden":false,"mode":"md"},{"id":937,"value":"chart = Tree(JSON.parse(JSON.stringify(hierarchy_tree_small)), {\n label: (d) => d.name,\n width: 1152,\n height: 1152,\n margin: 100\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1237,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":1233,"value":"Plot.plot({\n x: { label: \"Cosine similarity\" },\n y: { label: \"Weight\" },\n\n marks: [\n Plot.dot(links, {\n x: \"similarity\",\n y: \"old_weight\",\n tip: true,\n channels: { source: \"source\", target: \"target\" }\n }),\n Plot.ruleX([similarityThreshold])\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1075,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":1623,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":1080,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":1774,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":2247,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":541,"value":"viewof removeIIEGE = Inputs.toggle({\n label: \"Remove IIE GE lab (improves result)\",\n value: false\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":61,"value":"## Network","pinned":false,"hidden":false,"mode":"md"},{"id":63,"value":"nodes = labs.map(({ name, size, keywords, normalizedKeywords, ...data }) => ({\n id: name,\n size: Math.max(1, size + 1),\n topKeywords: normalizedKeywords.slice(0, 8).map((d) => d[0]),\n connectedLabs: Object.values(labLinks.get(name) ?? []).map(\n ({ source, target }) => (source == name ? target : source)\n ),\n normalizedKeywords,\n keywords,\n ...data\n}))","pinned":false,"hidden":false,"mode":"ojs"},{"id":688,"value":"topKeywordsByLab = d3\n .rollups(\n nodes.map((d) => d.normalizedKeywords.slice(0, 3)).flat(),\n (v) => d3.sum(v, (o) => o[1]),\n (d) => d[0]\n )\n .sort((a, b) => b[1] - a[1])","pinned":false,"hidden":false,"mode":"ojs"},{"id":295,"value":"labLinks = {\n let labLinks = new Map();\n\n for (let link of links) {\n if (!labLinks.has(link.source)) {\n labLinks.set(link.source, []);\n }\n if (!labLinks.has(link.target)) {\n labLinks.set(link.target, []);\n }\n labLinks.get(link.source).push(link);\n labLinks.get(link.target).push(link);\n }\n\n for (let [lab, links] of labLinks) {\n labLinks.set(\n lab,\n links.filter((d) => d.similarity > similarityThreshold)\n );\n }\n return labLinks;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1216,"value":"function cosineSimilarity(lab1, lab2) {\n let sum = 0, sum1 = 0, sum2 = 0;\n for(let keyword in lab1) {\n sum += lab1[keyword] * (lab2[keyword] || 0);\n sum1 += lab1[keyword] * lab1[keyword];\n }\n for(let keyword in lab2) {\n sum2 += lab2[keyword] * lab2[keyword];\n }\n return sum / (Math.sqrt(sum1) * Math.sqrt(sum2));\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":214,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":1252,"value":"links","pinned":false,"hidden":false,"mode":"ojs"},{"id":284,"value":"topLinks = [].concat(...Array.from(labLinks.values()))","pinned":false,"hidden":false,"mode":"ojs"},{"id":262,"value":"d3.extent(links, (d) => d.weight)","pinned":false,"hidden":false,"mode":"ojs"},{"id":280,"value":"graphHeight = 600","pinned":false,"hidden":false,"mode":"ojs"},{"id":1661,"value":"## Metaballs","pinned":false,"hidden":false,"mode":"md"},{"id":1674,"value":"dist = (a, b) => Math.sqrt((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1663,"value":"reduceCentroid = (prev, curr, i, array) => [\n prev[0] + curr[0] / array.length,\n prev[1] + curr[1] / array.length\n]","pinned":false,"hidden":false,"mode":"ojs"},{"id":1666,"value":"$39","pinned":false,"hidden":false,"mode":"ojs"},{"id":1669,"value":"function padHull(hull, padding = 30) {\n // from https://observablehq.com/@fil/hull-padding\n // <3 @Fil\n const h = hull;\n const n = h.length;\n\n const centroid = d3.polygonCentroid(h);\n return d3.polygonHull(\n h.map((p) =>\n d3.interpolate(\n p,\n centroid\n )(\n Math.min(\n 1,\n -padding / Math.hypot(centroid[0] - p[0], centroid[1] - p[1])\n )\n )\n )\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":891,"value":"## Concept tree","pinned":false,"hidden":false,"mode":"md"},{"id":777,"value":"concept_tree = {\n const json = await FileAttachment(\"tree.json\").json();\n\n // delete json.knowledge[\"social sciences\"];\n // delete json.knowledge[\"humanities\"];\n return json;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1115,"value":"function createTree(keywords, rootname = \"knowledge\", limit_depth = 5) {\n const mapWeights = new Map(keywords);\n\n var depth = 0;\n\n function toHierarchy(name, object) {\n if (!object) return {};\n const entries = Object.entries(object),\n isKeyword = name.includes(\"(keyword)\");\n if (isKeyword) name = name.replace(\" (keyword)\", \"\");\n let value = { name };\n if (isKeyword) value.value = mapWeights.get(name);\n if (entries.length > 0) {\n value.children = entries.map(([name, object]) => {\n depth++;\n return toHierarchy(name, object);\n });\n\n // value.children = value.children.filter((d) => d.value > 0);\n value.sum = d3.sum(value.children, (d) => d.value ?? d.sum ?? 0);\n\n if (depth >= limit_depth || value.children?.length == 0) {\n delete value.children;\n value.value = value.sum;\n }\n }\n value.depth = depth;\n depth--;\n return value;\n }\n\n return toHierarchy(rootname, concept_tree.knowledge);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1445,"value":"commonKeywordsTree = d3\n .rollups(\n labs\n .map((d) =>\n d.normalizedKeywords.map(([name, value]) => [\n name,\n value * fteWeighted ? value * d.size : value\n ])\n )\n .flat(),\n (v) => d3.sum(v, (d) => d[1]),\n (d) => d[0]\n )\n .sort((a, b) => b[1] - a[1])","pinned":false,"hidden":false,"mode":"ojs"},{"id":945,"value":"hierarchy_tree_small = createTree(commonKeywordsTree, \"knowledge\", 3)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1341,"value":"hierarchy_tree = createTree(commonKeywordsTree, \"knowledge\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":1823,"value":"conceptAncestor = new Map(\n d3_hierarchy_tree.children\n .map((concept) =>\n concept.descendants().map((d) => [d.data.name, concept.data.name])\n )\n .flat()\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1843,"value":"mainConcepts = hierarchy_tree.children.map((d) => d.name)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1890,"value":"hierarchy_tree_weights = {\n const hierarchy = d3.hierarchy(hierarchy_tree);\n const max = hierarchy.data.sum;\n const pairs = hierarchy\n .descendants()\n .map(({ data: { name, sum } }) => [name, sum]);\n\n return new Map(\n d3.rollups(\n pairs,\n (v) => d3.max(v.map((d) => d[1])),\n (d) => d[0]\n )\n );\n\n // .filter(([name, value]) => value > 0)\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1879,"value":"secondaryConcepts = hierarchy_tree.children\n .map((d) => d.children)\n .flat()\n .map((d) => [d.name, hierarchy_tree_weights.get(d.name)])\n .sort((a, b) => b[1] - a[1])","pinned":false,"hidden":false,"mode":"ojs"},{"id":1841,"value":"conceptColorScale = d3\n .scaleOrdinal()\n .domain(mainConcepts)\n .range(d3.schemeTableau10)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1320,"value":"d3_hierarchy_tree = d3.hierarchy(hierarchy_tree)","pinned":false,"hidden":false,"mode":"ojs"},{"id":778,"value":"concept_parent = {\n const json = await FileAttachment(\"concept_parent.json\").json(),\n entries = Object.entries(json);\n\n const keys = entries.flat();\n // return keys.filter((key) => keys.includes(key + \" (keyword)\"));\n // return Object.entries(json).filter(([parent, concept]) => parent);\n\n return new Map(entries);\n\n // const withoutKeywordStr = entries.map(([parent, concept]) => [\n // parent.replace(\" (keyword)\", \"\"),\n // concept\n // ]);\n\n // return new Map(withoutKeywordStr);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1529,"value":"function createSunburstWeights(lab) {\n const tree = createTree(lab.normalizedKeywords, lab.name);\n const hierarchy = d3.hierarchy(tree);\n const max = hierarchy.data.sum;\n return new Map(\n hierarchy\n .descendants()\n .map(({ data: { name, sum } }) => [name, sum / max])\n .filter(([name, value]) => value > 0)\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1734,"value":"new Set([...labs_trees.values()].map((d) => d.groups).flat())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1154,"value":"labs_trees = new Map([\n ...labs.map((d) => {\n const fullTree = createTree(d.normalizedKeywords, d.name);\n const groups = d3\n .hierarchy(fullTree)\n .descendants()\n .filter((d) => d.depth === 1 && d.data.sum > 0)\n .sort((a, b) => b.data.sum - a.data.sum)\n .map((d) => d.data)\n .map((d) => (d.children.length == 1 ? d.children[0] : d))\n .map((d) => (d.children.length == 1 ? d.children[0] : d))\n .filter((d) => d.sum > 1.3)\n .map((d) => d.name)\n .slice(0, 2);\n\n return [\n d.name,\n {\n full: fullTree,\n small: createTree(d.normalizedKeywords, d.name, 3),\n sunburst_weights: createSunburstWeights(d),\n sunburst: createSunburst(createTree(d.normalizedKeywords, d.name, 4)),\n groups\n }\n ];\n }),\n [\n \"default\",\n {\n full: {},\n small: {},\n sunburst: createSunburst(createTree([], \"No lab selected\"))\n }\n ]\n])","pinned":false,"hidden":false,"mode":"ojs"},{"id":547,"value":"## Co occurences","pinned":false,"hidden":false,"mode":"md"},{"id":578,"value":"$3a","pinned":false,"hidden":false,"mode":"ojs"},{"id":606,"value":"keywordList = [...new Set(topOverallKeywords.map((d) => d[0]))] // to map keywords to indices","pinned":false,"hidden":false,"mode":"ojs"},{"id":609,"value":"wordWeightExtent = d3.extent(wordLinks, (d) => d.weight)","pinned":false,"hidden":false,"mode":"ojs"},{"id":647,"value":"keywordLinks = {\n let keywordLinks = new Map();\n\n for (let link of wordLinks) {\n if (!keywordLinks.has(link.source)) {\n keywordLinks.set(link.source, []);\n }\n if (!keywordLinks.has(link.target)) {\n keywordLinks.set(link.target, []);\n }\n keywordLinks.get(link.source).push(link);\n keywordLinks.get(link.target).push(link);\n }\n\n for (let [lab, links] of keywordLinks) {\n links.sort((a, b) => b.weight - a.weight);\n keywordLinks.set(lab, links);\n }\n return keywordLinks;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":759,"value":"mutable hoveredKeywords = []","pinned":false,"hidden":false,"mode":"ojs"},{"id":599,"value":"wordLinks = {\n // initialize an empty array to store the links\n let links = [];\n\n // iterate through each pair of keywords\n for (let i = 0; i < matrix.length; i++) {\n for (let j = i + 1; j < matrix[i].length; j++) {\n // skip the diagonal and lower triangular part\n // if the matrix entry is nonzero, create a link\n if (matrix[i][j] > 0.3) {\n links.push({\n source: keywordList[i], // use the keyword list to get the keyword corresponding to the index\n target: keywordList[j],\n weight: matrix[i][j]\n });\n }\n }\n }\n\n return links.sort((a, b) => b.weight - a.weight);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":682,"value":"commonKeywords =d3\n .rollups(\n labs.map((d) => d.normalizedKeywords).flat(),\n (v) => d3.sum(v, (d) => d[1]),\n (d) => d[0]\n )\n .sort((a, b) => b[1] - a[1])","pinned":false,"hidden":false,"mode":"ojs"},{"id":549,"value":"topOverallKeywords = commonKeywords.slice(0, 200).concat(topKeywordsByLab)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1000,"value":"## Zoomable sunburst","pinned":false,"hidden":false,"mode":"md"},{"id":1008,"value":"partition = (data) => {\n const root = d3\n .hierarchy(data)\n .sum((d) => d.value)\n .sort((a, b) => b.value - a.value);\n return d3.partition().size([2 * Math.PI, root.height + 1])(root);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1011,"value":"color = d3.scaleOrdinal(\n d3.quantize(d3.interpolateRainbow, hierarchy_tree_small.children.length + 1)\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1014,"value":"format = d3.format(\",d\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":1017,"value":"radius = width / 6","pinned":false,"hidden":false,"mode":"ojs"},{"id":1020,"value":"arc = d3\n .arc()\n .startAngle((d) => d.x0)\n .endAngle((d) => d.x1)\n .padAngle((d) => Math.min((d.x1 - d.x0) / 2, 0.005))\n .padRadius(radius * 1.5)\n .innerRadius((d) => d.y0 * radius)\n .outerRadius((d) => Math.max(d.y0 * radius, d.y1 * radius - 1))","pinned":false,"hidden":false,"mode":"ojs"},{"id":41,"value":"## Data","pinned":false,"hidden":false,"mode":"md"},{"id":49,"value":"sizes = labs.map(({ name, size }) => ({\n lab: name,\n size\n}))","pinned":false,"hidden":false,"mode":"ojs"},{"id":106,"value":"$3b","pinned":false,"hidden":false,"mode":"ojs"},{"id":77,"value":"labs = wrangled_data","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"## Imports","pinned":false,"hidden":false,"mode":"md"},{"id":2189,"value":"import { slide, style } from \"@mbostock/slide\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":983,"value":"import {Pack} from \"@d3/pack\"\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":939,"value":"import { Tree } from \"@d3/radial-tree\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":1776,"value":"d3Cloud = require(\"d3-cloud@1\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":1230,"value":"import {ForceGraph} from \"@d3/force-directed-graph\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":2003,"value":"import { height } from \"@fil/height\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W3c","stdlib":"1","titling":"auto"}