30:T938,# Confusion Matrix: Persona-based iteration experiments I'm taking on this Persona: **Experienced Clinician** Background: 15 years in radiology. Uses AI tools daily but notices their diagnostic skills have become rusty when AI isn't available. **Their key concerns:** "I'm starting to rely too heavily on the AI flagging things for me. When the system is down, I feel less confident in my own judgment. Am I losing my edge?" **Their trust issues:** They've become dependent on AI recommendations and worry they're not maintaining their independent diagnostic skills. The high accuracy rates make the AI seem infallible, leading to complacency. **Design challenge:** How do you highlight AI uncertainty in ways that keep clinicians actively engaged in the diagnostic process, rather than passively accepting AI recommendations? ------------------------------------------------- Instead of consuming aggregated numbers, I'm going to look at the data prior to the rendering of the confusion matrix (for these experiments, my confusion matrix does not extend past a 2×2 grid). To get a better understanding and use my diagnostic thinking, I'll examine variance outside of a tabular view. I will use a sample of 100 patients for ease of math checking. **To counter this passive acceptance, we need to make visible what the aggregated metrics hide: the moments where the model's confidence diverges from reality.** ## Before the 2×2 grid The confusion matrix doesn't appear fully formed. It emerges from a fundamental comparison: what the model predicted versus what actually happened. My experiment steps back to understand this split before collapsing it into the familiar 2×2 grid. This visualization shows the first step in that process—separating patients into two streams based on the model's decision at your chosen threshold. On the left, everyone the model flagged as "YES" (high risk). On the right, everyone who actually had the condition. The connecting lines reveal where these streams align and where they diverge. The mismatches—those FP and FN counts—are what the experienced clinician needs to understand. Not just as numbers in a grid, but as real decision points where the model's certainty diverges from reality. The question can be anything. I'm using "Does the patient have cancer?"—so mismatches become critical here.31:T57f,#### Understanding the Confusion Matrix Split This visualisation shows how ${patients.length} patients are categorised at your chosen threshold of **${threshold.toFixed(2)}**. On the **left**, we see the model's predictions: **${confusionData.modelYes} patients flagged as YES** (high risk) and **${confusionData.modelNo} as NO** (low risk). On the **right**, the unchanging ground truth: **${confusionData.realYes} patients who actually have the condition** and **${confusionData.realNo} who don't**. The connecting lines reveal where prediction meets reality. The **top line** shows the ${confusionData.modelYes} patients the model said YES to: **${confusionData.TP} were correct** (true positives) but **${confusionData.FP} were false alarms** (false positives)—that's **${confusionData.FP} mismatch**. The **bottom line** shows the ${confusionData.modelNo} patients the model said NO to: **${confusionData.TN} were correctly cleared** (true negatives) but **${confusionData.FN} were missed cases** (false negatives)—**${confusionData.FN} mismatch**. Move the threshold slider and watch the trade-off: lower thresholds catch more cases (${confusionData.TP} of ${confusionData.realYes}) but create more false alarms (${confusionData.FP}). Higher thresholds reduce false alarms but risk missing real cases. The total mismatches? **${confusionData.mismatch} patients** where the model got it wrong.32:T5dc,## Clustering Patients: Seeing the Groups Before the Grid Now we can see all ${patients.length} patients grouped by their outcomes at threshold **${threshold.toFixed(2)}**. Each circle represents one person—not an abstraction, but an individual whose diagnosis depends on where the model placed them relative to that threshold line. The four clusters reveal the model's performance at a glance: - **${patientsByOutcome.TP.length} True Positives (TP)**: Correctly identified as high-risk. The model caught these cases. - **${patientsByOutcome.FN.length} False Negatives (FN)**: Missed cases. These patients have cancer but scored below the threshold. The model failed to flag them—potentially the most dangerous error. - **${patientsByOutcome.FP.length} False Positives (FP)**: False alarms. These patients don't have cancer but were flagged as high-risk, leading to unnecessary anxiety, further testing, and potential interventions. - **${patientsByOutcome.TN.length} True Negatives (TN)**: Correctly identified as low-risk. The model appropriately cleared them. **The mismatches matter most.** Those ${patientsByOutcome.FN.length} circles in FN represent people who left without intervention despite having cancer. Those ${patientsByOutcome.FP.length} circles in FP represent people subjected to the stress and procedures of a false cancer scare. Each circle is a person, and the clustering shows us the scale of these consequences before we collapse everything into a 2×2 grid of numbers. 33:T538,--- ## Circling Back: Active Engagement, Not Passive Consumption By deconstructing the confusion matrix—from split views to individual marks, **each representing a real person**—we've transformed a static metric into an interactive journey. Using dumbbells, clustered circles, and filters, we make the human scale visible. Each visualisation invites interpretation: examining threshold distances, comparing confidence levels, **seeing individual patients, not just aggregated numbers**. **Taking This Further:** - Adding patient context to each circle (age, risk factors, prior scans) - Dynamic filtering to isolate subgroups and reveal patterns - Advanced clustering to uncover systematic biases - Temporal views showing how patients move between categories The goal isn't to replace the confusion matrix—it's to prevent passive consumption. By making the journey from individuals to aggregated metrics visible and interactive, we keep clinicians **engaged in diagnostic thinking**. They're not just trusting the AI's output; they're understanding its reasoning, questioning edge cases, and maintaining their diagnostic edge. **The confusion matrix doesn't have to appear fully formed. By showing how it emerges, clinicians aren't just consuming metrics—they're maintaining that diagnostic edge they feared losing.** --- 34:T6f4,function dumbbellGrid(data, { threshold = 0.5, modelCol = '#fcb315', actualCol = '#fcb315', bgdCol = '#f8f8f8', lineCol = '#fcb315', thresholdCol = '#fcb315' } = {}) { // Create container div with flex layout const container = html`

`; // Add legend at the top (full width) const legend = html`

model

actual

threshold ${threshold.toFixed(2)}

`; container.appendChild(legend); // Create individual dumbbell for each patient data.forEach(d => { const patientDiv = html`

${createDumbbell(d, { threshold, modelCol, actualCol, bgdCol, lineCol, thresholdCol })}

`; container.appendChild(patientDiv); }); return container; }35:Tbff,function createDumbbell(data, { threshold = 0.5, modelCol = '#fcb315', actualCol = '#fcb315', bgdCol = '#f8f8f8', lineCol = '#fcb315', thresholdCol = '#fcb315', width = 180, height = 200 } = {}) { const margin = {left: 40, top: 20, right: 40, bottom: 40}; const plotHeight = height - margin.top - margin.bottom; const yScale = d3.scaleLinear() .domain([0, 1]) .range([plotHeight, 0]); const x = width / 2; const modelY = yScale(data.values[0]) + margin.top; const actualY = yScale(data.values[1]) + margin.top; const thresholdY = yScale(threshold) + margin.top; const svg = DOM.svg(width, height); const sel = d3.select(svg); // Background with rounded corners for block effect sel.append('rect') .attr('width', width) .attr('height', height) .attr('fill', 'transparent'); // Vertical reference line sel.append('line') .attr('x1', x) .attr('x2', x) .attr('y1', margin.top) .attr('y2', height - margin.bottom) .attr('stroke', '#e0e0e0') .attr('stroke-width', 1); // Threshold line sel.append('line') .attr('x1', margin.left - 10) .attr('x2', width - margin.right + 10) .attr('y1', thresholdY) .attr('y2', thresholdY) .attr('stroke', thresholdCol) .attr('stroke-width', 2) .attr('stroke-dasharray', '5,5'); sel.append('text') .attr('x', margin.left - 15) .attr('y', thresholdY + 4) .attr('text-anchor', 'end') .style('font-size', '11px') .style('fill', thresholdCol) .text(threshold.toFixed(1)); // Connecting line (dumbbell bar) sel.append('line') .attr('x1', x) .attr('x2', x) .attr('y1', modelY) .attr('y2', actualY) .attr('stroke', lineCol) .attr('stroke-width', 6); // Model circle (hollow) sel.append('circle') .attr('cx', x) .attr('cy', modelY) .attr('r', 14) .attr('fill', '#ffffff') .attr('stroke', modelCol) .attr('stroke-width', 3); // Model score label sel.append('text') .attr('x', x + 30) .attr('y', modelY + 5) .style('font-size', '14px') .style('font-weight', 'bold') .text(data.values[0].toFixed(2)); // Actual circle (filled) sel.append('circle') .attr('cx', x) .attr('cy', actualY) .attr('r', 14) .attr('fill', actualCol); // Actual Y/N label sel.append('text') .attr('x', x) .attr('y', actualY + 6) .attr('text-anchor', 'middle') .style('font-size', '16px') .style('font-weight', 'bold') .style('fill', '#000') .text(data.values[1] === 1 ? 'Y' : 'N'); // Patient ID and outcome label at bottom sel.append('text') .attr('x', x + 70) .attr('y', height - margin.bottom + 40) .attr('text-anchor', 'middle') .style('font-size', '12px') .style('fill', '#dd6464') .text(`#${data.id}`); sel.append('text') .attr('x', x) .attr('y', height - margin.bottom + 35) .attr('text-anchor', 'middle') .style('font-size', '18px') .style('font-weight', 'bold') .text(data.label); return svg; }36:T22e0,function confusionSplit(data, { width = 500, height = 300, margin = {left: 60, top: 100, right: 60, bottom: 40}, bgdCol = '#fff', boxFill = '#f27291', textCol = '#333', countCol = '#111', lineCol = '#fcb315', mismatchTextCol = '#f27291', labelBoxCol = '#fcb315', labelTextCol = '#333', tpCol = '#fcb315', fpCol = '#fcb315', fnCol = '#fcb315', tnCol = '#fcb315' } = {}) { const w = width + margin.left + margin.right; const h = height + margin.top + margin.bottom; const svg = DOM.svg(w, h); const sel = d3.select(svg); // Background sel.append('rect') .attr('width', w) .attr('height', h) .attr('fill', bgdCol); // Define box dimensions const boxWidth = 140; const boxHeight = 140; const labelHeight = 35; const leftX = margin.left; const rightX = w - margin.right - boxWidth; const centerY = margin.top + 30; const midX = (leftX + boxWidth + rightX) / 2; // Left side: Model Predicts const modelGroup = sel.append('g') .attr('transform', `translate(${leftX}, ${centerY})`); // Model box (square, no border) modelGroup.append('rect') .attr('width', boxWidth) .attr('height', boxHeight) .attr('fill', boxFill); modelGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight / 2 - 10) .attr('text-anchor', 'middle') // .style('font-size', '14px') .style('fill', textCol) .text('model'); modelGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight / 2 + 10) .attr('text-anchor', 'middle') // .style('font-size', '14px') .style('fill', textCol) .text('predicts'); // YES label (top) - yellow box, no border const yesTopGroup = modelGroup.append('g') .attr('transform', `translate(${boxWidth / 2}, ${-labelHeight})`); yesTopGroup.append('rect') .attr('x', -25) .attr('y', 0) .attr('width', 50) .attr('height', 28) .attr('fill', labelBoxCol); yesTopGroup.append('text') .attr('text-anchor', 'middle') .attr('y', 19) // .style('font-size', '14px') .style('font-weight', '600') .style('fill', labelTextCol) .text('YES'); // Line connecting YES to box modelGroup.append('line') .attr('x1', boxWidth / 2) .attr('y1', -labelHeight + 28) .attr('x2', boxWidth / 2) .attr('y2', 0) .attr('stroke', labelBoxCol) .attr('stroke-width', 5); // Count above YES modelGroup.append('text') .attr('x', boxWidth / 2) .attr('y', -labelHeight - 8) .attr('text-anchor', 'middle') // .style('font-size', '12px') .style('fill', countCol) .text(data.modelYes); // NO label (bottom) - yellow box, no border const noBottomGroup = modelGroup.append('g') .attr('transform', `translate(${boxWidth / 2}, ${boxHeight + 5})`); noBottomGroup.append('rect') .attr('x', -20) .attr('y', 0) .attr('width', 40) .attr('height', 28) .attr('fill', labelBoxCol); noBottomGroup.append('text') .attr('text-anchor', 'middle') .attr('y', 19) // .style('font-size', '14px') .style('font-weight', '600') .style('fill', labelTextCol) .text('NO'); // Line connecting box to NO modelGroup.append('line') .attr('x1', boxWidth / 2) .attr('y1', boxHeight) .attr('x2', boxWidth / 2) .attr('y2', boxHeight + 5) .attr('stroke', labelBoxCol) .attr('stroke-width', 5); // Count below NO modelGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight + labelHeight + 20) .attr('text-anchor', 'middle') // .style('font-size', '12px') .style('fill', countCol) .text(data.modelNo); // Right side: Real Results const realGroup = sel.append('g') .attr('transform', `translate(${rightX}, ${centerY})`); // Real box (square, no border) realGroup.append('rect') .attr('width', boxWidth) .attr('height', boxHeight) .attr('fill', boxFill); realGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight / 2 - 10) .attr('text-anchor', 'middle') // .style('font-size', '14px') .style('fill', textCol) .text('real'); realGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight / 2 + 10) .attr('text-anchor', 'middle') // .style('font-size', '14px') .style('fill', textCol) .text('results'); // YES label (top right) - yellow box, no border const yesTopRightGroup = realGroup.append('g') .attr('transform', `translate(${boxWidth / 2}, ${-labelHeight})`); yesTopRightGroup.append('rect') .attr('x', -25) .attr('y', 0) .attr('width', 50) .attr('height', 28) .attr('fill', labelBoxCol); yesTopRightGroup.append('text') .attr('text-anchor', 'middle') .attr('y', 19) // .style('font-size', '14px') .style('font-weight', '600') .style('fill', labelTextCol) .text('YES'); // Line connecting YES to box realGroup.append('line') .attr('x1', boxWidth / 2) .attr('y1', -labelHeight + 28) .attr('x2', boxWidth / 2) .attr('y2', 0) .attr('stroke', labelBoxCol) .attr('stroke-width', 5); // Count above YES realGroup.append('text') .attr('x', boxWidth / 2) .attr('y', -labelHeight - 8) .attr('text-anchor', 'middle') // .style('font-size', '12px') .style('fill', countCol) .text(data.realYes); // NO label (bottom right) - yellow box, no border const noBottomRightGroup = realGroup.append('g') .attr('transform', `translate(${boxWidth / 2}, ${boxHeight + 5})`); noBottomRightGroup.append('rect') .attr('x', -20) .attr('y', 0) .attr('width', 40) .attr('height', 28) .attr('fill', labelBoxCol); noBottomRightGroup.append('text') .attr('text-anchor', 'middle') .attr('y', 19) // .style('font-size', '14px') .style('font-weight', '600') .style('fill', labelTextCol) .text('NO'); // Line connecting box to NO realGroup.append('line') .attr('x1', boxWidth / 2) .attr('y1', boxHeight) .attr('x2', boxWidth / 2) .attr('y2', boxHeight + 5) .attr('stroke', labelBoxCol) .attr('stroke-width', 5); // Count below NO realGroup.append('text') .attr('x', boxWidth / 2) .attr('y', boxHeight + labelHeight + 20) .attr('text-anchor', 'middle') // .style('font-size', '12px') .style('fill', countCol) .text(data.realNo); // Top connection line (TP/FP line) - yellow dotted const topLineY = centerY - labelHeight / 2 - 28; sel.append('line') .attr('x1', leftX + boxWidth - 20) .attr('y1', topLineY) .attr('x2', rightX + 20) .attr('y2', topLineY) .attr('stroke', lineCol) .attr('stroke-width', 1) .attr('stroke-dasharray', '3,3'); // TP label (top left of line) - black sel.append('text') .attr('x', leftX + boxWidth + 15) .attr('y', topLineY - 5) .attr('text-anchor', 'start') // .style('font-size', '11px') .style('font-weight', '600') .style('fill', tpCol) .text(`TP: ${data.TP || 0}`); // FP label (top right of line) - black sel.append('text') .attr('x', rightX - 15) .attr('y', topLineY - 5) .attr('text-anchor', 'end') // .style('font-size', '11px') .style('font-weight', '600') .style('fill', fpCol) .text(`FP: ${data.FP || 0}`); // FP mismatch text (centered below top line) - red sel.append('text') .attr('x', midX) .attr('y', topLineY + 30) .attr('text-anchor', 'middle') // .style('font-size', '12px') .style('font-weight', '600') .style('fill', mismatchTextCol) .text(`${data.FP} mismatch`); // Bottom connection line (FN line) - yellow dotted, moved up const bottomLineY = centerY + boxHeight + labelHeight + 22; sel.append('line') .attr('x1', leftX + boxWidth - 20) .attr('y1', bottomLineY) .attr('x2', rightX + 20) .attr('y2', bottomLineY) .attr('stroke', lineCol) .attr('stroke-width', 1) .attr('stroke-dasharray', '3,3'); // TN label (bottom left of line) sel.append('text') .attr('x', leftX + boxWidth + 15) .attr('y', bottomLineY - 5) .attr('text-anchor', 'start') // .style('font-size', '11px') .style('font-weight', '600') .style('fill', tnCol) .text(`TN: ${data.TN || 0}`); // FN label (bottom right of line) - black sel.append('text') .attr('x', rightX - 15) .attr('y', bottomLineY - 5) .attr('text-anchor', 'end') // .style('font-size', '11px') .style('font-weight', '600') .style('fill', fnCol) .text(`FN: ${data.FN || 0}`); // FN mismatch text (centered below bottom line) - red sel.append('text') .attr('x', midX) .attr('y', bottomLineY + 30) .attr('text-anchor', 'middle') .style('font-weight', '600') // .style('font-size', '12px') .style('fill', mismatchTextCol) .text(`${data.FN} mismatch`); return svg; }37:Td77,function clusterDiagram(data, { width = 900, height = 700, margin = {top: 60, right: 60, bottom: 60, left: 60}, bgdCol = '#ffffff', tpCol = '#fcb315', tnCol = '#dd6464', fpCol = '#dd6464', fnCol = '#fcb315', circleRadius = 8, circleSpacing = 3, // Consistent spacing between circles clusterPadding = 100 } = {}) { const w = width + margin.left + margin.right; const h = height + margin.top + margin.bottom; const svg = DOM.svg(w, h); const sel = d3.select(svg); // Background sel.append('rect') .attr('width', w) .attr('height', h) .attr('fill', bgdCol); // Calculate cluster positions (2x2 grid with more spacing) const clusterWidth = (width - margin.left - margin.right - clusterPadding) / 2; const clusterHeight = (height - margin.top - margin.bottom - clusterPadding) / 2; const clusters = [ {label: 'TP', x: margin.left + clusterWidth / 2, y: margin.top + clusterHeight / 2, col: tpCol, fill: tpCol}, {label: 'FN', x: margin.left + clusterWidth * 1.5 + clusterPadding, y: margin.top + clusterHeight / 2, col: fnCol, fill: 'none'}, {label: 'FP', x: margin.left + clusterWidth / 2, y: margin.top + clusterHeight * 1.5 + clusterPadding, col: fpCol, fill: 'none'}, {label: 'TN', x: margin.left + clusterWidth * 1.5 + clusterPadding, y: margin.top + clusterHeight * 1.5 + clusterPadding, col: tnCol, fill: tnCol} ]; // Group data by outcome const grouped = {TP: [], FN: [], FP: [], TN: []}; data.forEach(d => grouped[d.outcome].push(d)); // Draw each cluster clusters.forEach(cluster => { const patients = grouped[cluster.label]; const count = patients.length; // Calculate the size needed for pack layout based on consistent spacing // The diameter of each packed circle is circleRadius * 2 + circleSpacing const packedCircleSize = (circleRadius * 2 + circleSpacing * 2); const packSize = Math.sqrt(count) * packedCircleSize * 1.5; // Calculate cluster circle radius to contain the packed circles const clusterRadius = Math.max(60, packSize / 2); // Draw enclosing circle sel.append('circle') .attr('cx', cluster.x) .attr('cy', cluster.y) .attr('r', clusterRadius) .attr('fill', 'none') .attr('stroke', cluster.col) .attr('stroke-width', 2) .attr('stroke-dasharray', '5,5'); // Label sel.append('text') .attr('x', cluster.x) .attr('y', cluster.y - clusterRadius - 15) .attr('text-anchor', 'middle') .style('font-size', '18px') .style('font-weight', 'bold') .style('fill', cluster.col) .text(cluster.label); // Pack patient circles inside with consistent spacing const packLayout = d3.pack() .size([packSize, packSize]) .padding(circleSpacing) // Use consistent padding .radius(() => circleRadius); // Force consistent radius const root = d3.hierarchy({children: patients}) .sum(d => 1); const nodes = packLayout(root).leaves(); // Draw patient circles nodes.forEach(node => { sel.append('circle') .attr('cx', cluster.x + node.x - packSize / 2) .attr('cy', cluster.y + node.y - packSize / 2) .attr('r', circleRadius) .attr('fill', cluster.fill === 'none' ? '#ffffff' : cluster.fill) .attr('stroke', cluster.col) .attr('stroke-width', 2); }); }); return svg; }38:T167b,// Static patient dataset - exactly 100 cases // created by CLAUDE // At threshold 0.5: 20 TP, 5 FP, 10 FN, 65 TN patients = [ // TRUE POSITIVES (20): actualPositive=true, riskScore >= 0.5 { id: 1, actualPositive: true, riskScore: 0.95 }, { id: 2, actualPositive: true, riskScore: 0.92 }, { id: 3, actualPositive: true, riskScore: 0.88 }, { id: 4, actualPositive: true, riskScore: 0.85 }, { id: 5, actualPositive: true, riskScore: 0.82 }, { id: 6, actualPositive: true, riskScore: 0.78 }, { id: 7, actualPositive: true, riskScore: 0.75 }, { id: 8, actualPositive: true, riskScore: 0.71 }, { id: 9, actualPositive: true, riskScore: 0.68 }, { id: 10, actualPositive: true, riskScore: 0.65 }, { id: 11, actualPositive: true, riskScore: 0.62 }, { id: 12, actualPositive: true, riskScore: 0.59 }, { id: 13, actualPositive: true, riskScore: 0.56 }, { id: 14, actualPositive: true, riskScore: 0.53 }, { id: 15, actualPositive: true, riskScore: 0.51 }, { id: 16, actualPositive: true, riskScore: 0.50 }, { id: 17, actualPositive: true, riskScore: 0.52 }, { id: 18, actualPositive: true, riskScore: 0.54 }, { id: 19, actualPositive: true, riskScore: 0.57 }, { id: 20, actualPositive: true, riskScore: 0.60 }, // FALSE NEGATIVES (10): actualPositive=true, riskScore < 0.5 { id: 21, actualPositive: true, riskScore: 0.48 }, { id: 22, actualPositive: true, riskScore: 0.45 }, { id: 23, actualPositive: true, riskScore: 0.42 }, { id: 24, actualPositive: true, riskScore: 0.38 }, { id: 25, actualPositive: true, riskScore: 0.35 }, { id: 26, actualPositive: true, riskScore: 0.31 }, { id: 27, actualPositive: true, riskScore: 0.28 }, { id: 28, actualPositive: true, riskScore: 0.24 }, { id: 29, actualPositive: true, riskScore: 0.20 }, { id: 30, actualPositive: true, riskScore: 0.15 }, // FALSE POSITIVES (5): actualPositive=false, riskScore >= 0.5 { id: 31, actualPositive: false, riskScore: 0.72 }, { id: 32, actualPositive: false, riskScore: 0.64 }, { id: 33, actualPositive: false, riskScore: 0.58 }, { id: 34, actualPositive: false, riskScore: 0.53 }, { id: 35, actualPositive: false, riskScore: 0.51 }, // TRUE NEGATIVES (65): actualPositive=false, riskScore < 0.5 { id: 36, actualPositive: false, riskScore: 0.48 }, { id: 37, actualPositive: false, riskScore: 0.46 }, { id: 38, actualPositive: false, riskScore: 0.44 }, { id: 39, actualPositive: false, riskScore: 0.42 }, { id: 40, actualPositive: false, riskScore: 0.40 }, { id: 41, actualPositive: false, riskScore: 0.38 }, { id: 42, actualPositive: false, riskScore: 0.36 }, { id: 43, actualPositive: false, riskScore: 0.34 }, { id: 44, actualPositive: false, riskScore: 0.32 }, { id: 45, actualPositive: false, riskScore: 0.30 }, { id: 46, actualPositive: false, riskScore: 0.28 }, { id: 47, actualPositive: false, riskScore: 0.26 }, { id: 48, actualPositive: false, riskScore: 0.24 }, { id: 49, actualPositive: false, riskScore: 0.22 }, { id: 50, actualPositive: false, riskScore: 0.20 }, { id: 51, actualPositive: false, riskScore: 0.18 }, { id: 52, actualPositive: false, riskScore: 0.16 }, { id: 53, actualPositive: false, riskScore: 0.14 }, { id: 54, actualPositive: false, riskScore: 0.12 }, { id: 55, actualPositive: false, riskScore: 0.10 }, { id: 56, actualPositive: false, riskScore: 0.09 }, { id: 57, actualPositive: false, riskScore: 0.08 }, { id: 58, actualPositive: false, riskScore: 0.07 }, { id: 59, actualPositive: false, riskScore: 0.06 }, { id: 60, actualPositive: false, riskScore: 0.05 }, { id: 61, actualPositive: false, riskScore: 0.04 }, { id: 62, actualPositive: false, riskScore: 0.03 }, { id: 63, actualPositive: false, riskScore: 0.02 }, { id: 64, actualPositive: false, riskScore: 0.01 }, { id: 65, actualPositive: false, riskScore: 0.47 }, { id: 66, actualPositive: false, riskScore: 0.45 }, { id: 67, actualPositive: false, riskScore: 0.43 }, { id: 68, actualPositive: false, riskScore: 0.41 }, { id: 69, actualPositive: false, riskScore: 0.39 }, { id: 70, actualPositive: false, riskScore: 0.37 }, { id: 71, actualPositive: false, riskScore: 0.35 }, { id: 72, actualPositive: false, riskScore: 0.33 }, { id: 73, actualPositive: false, riskScore: 0.31 }, { id: 74, actualPositive: false, riskScore: 0.29 }, { id: 75, actualPositive: false, riskScore: 0.27 }, { id: 76, actualPositive: false, riskScore: 0.25 }, { id: 77, actualPositive: false, riskScore: 0.23 }, { id: 78, actualPositive: false, riskScore: 0.21 }, { id: 79, actualPositive: false, riskScore: 0.19 }, { id: 80, actualPositive: false, riskScore: 0.17 }, { id: 81, actualPositive: false, riskScore: 0.15 }, { id: 82, actualPositive: false, riskScore: 0.13 }, { id: 83, actualPositive: false, riskScore: 0.11 }, { id: 84, actualPositive: false, riskScore: 0.49 }, { id: 85, actualPositive: false, riskScore: 0.47 }, { id: 86, actualPositive: false, riskScore: 0.45 }, { id: 87, actualPositive: false, riskScore: 0.43 }, { id: 88, actualPositive: false, riskScore: 0.41 }, { id: 89, actualPositive: false, riskScore: 0.39 }, { id: 90, actualPositive: false, riskScore: 0.37 }, { id: 91, actualPositive: false, riskScore: 0.35 }, { id: 92, actualPositive: false, riskScore: 0.33 }, { id: 93, actualPositive: false, riskScore: 0.31 }, { id: 94, actualPositive: false, riskScore: 0.29 }, { id: 95, actualPositive: false, riskScore: 0.27 }, { id: 96, actualPositive: false, riskScore: 0.25 }, { id: 97, actualPositive: false, riskScore: 0.23 }, { id: 98, actualPositive: false, riskScore: 0.21 }, { id: 99, actualPositive: false, riskScore: 0.19 }, { id: 100, actualPositive: false, riskScore: 0.17 } ]39:[{"id":"b5f342a26d459595342974995748322f19c8ce1cc7fa1eefaf6fa4207b17c0c1951854313830c7b095beadaa43d0b2ee6fe596348de171c41794ffc9dcc227c1","href":"https://static.observableusercontent.com/files/b5f342a26d459595342974995748322f19c8ce1cc7fa1eefaf6fa4207b17c0c1951854313830c7b095beadaa43d0b2ee6fe596348de171c41794ffc9dcc227c1","name":"confusion-matrix-key-cluster.png","type":"image/png","lastModified":1760000198179.409,"size":5398},{"id":"bf4613cddffe279d865d753839e86bd3efa20781a58ee5a5c5e982aeb0400107acf6e21e18cf669419d75a73f63fc722251573ad6433d5cf99a0740fe9cefe65","href":"https://static.observableusercontent.com/files/bf4613cddffe279d865d753839e86bd3efa20781a58ee5a5c5e982aeb0400107acf6e21e18cf669419d75a73f63fc722251573ad6433d5cf99a0740fe9cefe65","name":"confusion-matrix-outputs@1.png","type":"image/png","lastModified":1759939781163.35,"size":14493},{"id":"214106cf193bc5fd49b2ca4c05fbfa57f884474bc8bad78bf44c77db9c14322aa2f9d317255059cfc8053cb50dbeb23e82e7d38f2fcf8b6d01e319ff83833e2a","href":"https://static.observableusercontent.com/files/214106cf193bc5fd49b2ca4c05fbfa57f884474bc8bad78bf44c77db9c14322aa2f9d317255059cfc8053cb50dbeb23e82e7d38f2fcf8b6d01e319ff83833e2a","name":"confusion-matrix-ideas-range-bar@3.png","type":"image/png","lastModified":1759954805404.167,"size":24183}] 2e:{"title":"Confusion Matrix: Persona-based iteration experiments","cells":[{"id":133,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":384,"value":"//data \npatients","pinned":true,"hidden":false,"mode":"ojs"},{"id":113,"value":"confusionSplit(confusionData)","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"//interactive threshold slider \nviewof threshold = Inputs.range([0.1, 0.9], {\n value: 0.5, \n step: 0.01, \n label: \"Decision threshold\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":193,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":174,"value":"----------------------\n\nAs a clinical I want to know how close were the accurate answers, as well as the mismatches. This can help me revisit the threshold and datasets feeding the model. \n\nIf a patient has cancer, that is either true (yes) or false (no). \nWhen I add a prediction to that (a ML model) that also give a either true (yes) or false (no). \n\nNow when I compare these results we have four possible outputs per patient:\n\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":187,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":241,"value":"Before counting outcomes, let's examine individual patients. Each dumbbell bar shows a single patient's journey through the model's decision process. The hollow circle marks where the model placed this patient on the risk scale (0 to 1), while the filled circle shows the actual diagnosis—pinned at the top (1) for cancer present, or bottom (0) for cancer absent.\n\nThe threshold line (here at 0.5) acts as the decision boundary: above it, the model predicts YES; below it, NO. This creates our four possible outcomes.\n\nThe length of the dumbbell shows the gap between prediction and reality—but length alone doesn't tell the whole story. In true positives and true negatives, the model got it right regardless of confidence level. But in false positives and false negatives, the dumbbell reveals two types of error:\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":235,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"Close misses happen when the model's score is near the threshold but on the wrong side—like your FN example, where the model scored just below 0.5 but the patient actually had cancer. The dumbbell is long (showing the prediction-reality gap), but the hollow circle sits precariously close to being correct. These are the marginal calls where a **slight threshold adjustment could flip the outcome**.\n\nConfident mistakes show a wide gap with the model's score far from the threshold. A short dumbbell with the hollow circle distant from the threshold line means the model was confidently wrong—perhaps the most concerning type of error.\n\nFor experienced clinicians, this matters: a patient **hovering near the threshold might warrant additional scrutiny**, while someone far from the boundary represents a clearer (if potentially incorrect) signal.\n\n---------------------__","pinned":false,"hidden":false,"mode":"md"},{"id":314,"value":"---\n\n### Currently showing: **${outcomeFilter === \"all\" ? \"all patients\" : outcomeFilter === \"mismatches\" ? \"mismatches only (FP + FN)\" : outcomeFilter}**\n\nAt threshold **${threshold.toFixed(2)}**, you're viewing **${filteredPatients.length} patients** out of ${patients.length} total.\n","pinned":false,"hidden":false,"mode":"md"},{"id":247,"value":"// Dropdown selector for filtering\nviewof outcomeFilter = Inputs.select(\n [\"all\", \"mismatches\", \"TP\", \"FP\", \"FN\", \"TN\"],\n {\n label: \"Show patients:\",\n value: \"mismatches\"\n }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":254,"value":"dumbbellGrid(dumbbellData, {\n threshold: threshold\n})\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"## Individual Patient View: Understanding Each Decision\n\nBefore aggregating into the 2×2 confusion matrix, let's examine individual patients to understand how the model makes its calls. Each dumbbell shows one patient's journey: the **hollow circle** marks the model's risk score (0 to 1), while the **filled circle** shows the actual diagnosis—Y for cancer present, N for absent.\n\nThe **dumbbell length** reveals the prediction-reality gap. Short dumbbells in TP and TN show the model got it right. Longer dumbbells in FP and FN expose the errors:\n\n**Close misses** happen when the model's score sits just shy of being correct—hovering near the threshold but on the wrong side. These marginal calls might flip with a slight threshold adjustment.\n\n**Confident mistakes** show the model far from the threshold but wrong about the diagnosis. These represent clearer signals that led to incorrect conclusions.\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":291,"value":"--------------------------\n\nAs a clinician, this individual patient view has helped me understand the model's accuracy at a granular level. Now I want to see how these patients group together before they're aggregated into the 2×2 confusion matrix. \n\nWe can visualise all patients clustered by their four possible outcomes: **TP** (true positives), **FN** (false negatives), **FP** (false positives), and **TN** (true negatives). Each cluster shows the distribution of patients within that category, revealing patterns in how the model performs across different risk score ranges.","pinned":false,"hidden":false,"mode":"md"},{"id":296,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":328,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":325,"value":"clusterDiagram(clusterData)","pinned":true,"hidden":false,"mode":"ojs"},{"id":336,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":374,"value":"```\n ___ ___ _____ \n / /\\ /__/\\ / /::\\ \n / /:/_ \\ \\:\\ / /:/\\:\\ \n / /:/ /\\ \\ \\:\\ / /:/ \\:\\ \n / /:/ /:/_ _____\\__\\:\\ /__/:/ \\__\\:|\n /__/:/ /:/ /\\ /__/::::::::\\ \\ \\:\\ / /:/\n \\ \\:\\/:/ /:/ \\ \\:\\~~\\~~\\/ \\ \\:\\ /:/ \n \\ \\::/ /:/ \\ \\:\\ ~~~ \\ \\:\\/:/ \n \\ \\:\\/:/ \\ \\:\\ \\ \\::/ \n \\ \\::/ \\ \\:\\ \\__\\/ \n \\__\\/ \\__\\/ \n ```\n\n----------------","pinned":false,"hidden":false,"mode":"md"},{"id":289,"value":"------------------------------------------","pinned":false,"hidden":false,"mode":"md"},{"id":264,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":266,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":45,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":320,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":140,"value":"-----------------------------------\n\nbelow is my sample/dummy data and helper functions:\n\nnote: *claude has been used to generate the data and some helper functions*","pinned":false,"hidden":false,"mode":"md"},{"id":5,"value":"```\n _____ ___ ___ \n / /::\\ / /\\ ___ / /\\ \n / /:/\\:\\ / /::\\ / /\\ / /::\\ \n / /:/ \\:\\ / /:/\\:\\ / /:/ / /:/\\:\\ \n /__/:/ \\__\\:| / /:/~/::\\ / /:/ / /:/~/::\\ \n \\ \\:\\ / /:/ /__/:/ /:/\\:\\ / /::\\ /__/:/ /:/\\:\\\n \\ \\:\\ /:/ \\ \\:\\/:/__\\/ /__/:/\\:\\ \\ \\:\\/:/__\\/\n \\ \\:\\/:/ \\ \\::/ \\__\\/ \\:\\ \\ \\::/ \n \\ \\::/ \\ \\:\\ \\ \\:\\ \\ \\:\\ \n \\__\\/ \\ \\:\\ \\__\\/ \\ \\:\\ \n \\__\\/ \\__\\/ \n```","pinned":false,"hidden":false,"mode":"md"},{"id":322,"value":"clusterData = patients.map(p => ({\n id: p.id,\n outcome: outcomeFn(p, threshold)\n}))","pinned":false,"hidden":false,"mode":"ojs"},{"id":332,"value":"patientsByOutcome = {\n const grouped = { TP: [], FP: [], FN: [], TN: [] };\n \n patients.forEach(patient => {\n const outcome = outcomeFn(patient, threshold);\n grouped[outcome].push(patient);\n });\n \n return grouped;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":107,"value":"confusionData = {\n const groups = { TP: 0, FP: 0, FN: 0, TN: 0 };\n patients.forEach(p => {\n groups[outcomeFn(p, threshold)]++;\n });\n \n return {\n modelYes: groups.TP + groups.FP,\n modelNo: groups.FN + groups.TN,\n realYes: groups.TP + groups.FN,\n realNo: groups.FP + groups.TN,\n mismatch: groups.FP + groups.FN,\n TP: groups.TP,\n FP: groups.FP,\n FN: groups.FN,\n TN: groups.TN\n };\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":252,"value":"dumbbellData = filteredPatients.map(p => ({\n id: p.id,\n label: outcomeFn(p, threshold),\n values: [p.riskScore, p.actualPositive ? 1 : 0]\n}))\n\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":250,"value":"// Filter patients based on selection\nfilteredPatients = {\n const outcome = outcomeFilter;\n \n if (outcome === \"all\") {\n return patients;\n } else if (outcome === \"mismatches\") {\n return patients.filter(p => {\n const result = outcomeFn(p, threshold);\n return result === \"FP\" || result === \"FN\";\n });\n } else {\n return patients.filter(p => outcomeFn(p, threshold) === outcome);\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":18,"value":"$38","pinned":true,"hidden":false,"mode":"ojs"},{"id":34,"value":"// Helper function\noutcomeFn = (patient, threshold) => {\n const predicted = patient.riskScore >= threshold;\n const actual = patient.actualPositive;\n if (predicted && actual) return \"TP\";\n if (predicted && !actual) return \"FP\";\n if (!predicted && actual) return \"FN\";\n return \"TN\";\n}\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":118,"value":"```\n ___ ___ ___ \n / /\\ / /\\ / /\\ \n / /:/ / /:/_ / /:/_ \n / /:/ / /:/ /\\ / /:/ /\\ \n / /:/ ___ / /:/ /::\\ / /:/ /::\\ \n /__/:/ / /\\ /__/:/ /:/\\:\\ /__/:/ /:/\\:\\\n \\ \\:\\ / /:/ \\ \\:\\/:/~/:/ \\ \\:\\/:/~/:/\n \\ \\:\\ /:/ \\ \\::/ /:/ \\ \\::/ /:/ \n \\ \\:\\/:/ \\__\\/ /:/ \\__\\/ /:/ \n \\ \\::/ /__/:/ /__/:/ \n \\__\\/ \\__\\/ \\__\\/ \n ```","pinned":false,"hidden":false,"mode":"md"},{"id":116,"value":"

\n\n","pinned":true,"hidden":false,"mode":"html"}],"files":"$W39","stdlib":"1","titling":"auto"}