31:[] 2f:{"title":"How to plot audio recordings at any zoom level","cells":[{"id":0,"value":"# How to plot audio recordings at any zoom level\n\nI would like to write some notebooks about working with audio recordings. For short recordings, I can easily plot them using Observable Plot, but for longer samples, there are far too many points to get good performance. (One million points is only 20 seconds of audio at 44k samples per second.) In this notebook, I describe an algorithm for reducing the number of points to plot.\n\nThis algorithm works at a variety of zoom levels and is fast enough to zoom in and out interactively. (See the demo at the end.) When zoomed out, the audio envelope is displayed, and when you zoom in, it smoothly turns into the waveform.","pinned":false,"hidden":false,"mode":"md"},{"id":14,"value":"## Test data\n\nHere's an audio recording that's about a minute long. (${audioData.length} samples.)","pinned":false,"hidden":false,"mode":"md"},{"id":22,"value":"import { weltmeisterAchatBassMaster } from \"@skybrian/accordion-recordings\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"audioData = (\n await new window.AudioContext().decodeAudioData(\n await weltmeisterAchatBassMaster.arrayBuffer()\n )\n).getChannelData(0)","pinned":false,"hidden":false,"mode":"ojs"},{"id":42,"value":"## Finding the envelope\n\nThe envelope of an audio sample roughly shows its loudness. It's often useful for adjusting volume and finding different parts of a sample visually. The relevant points will be local maximums and minimums on the audio waveform. So, let's start by finding those:","pinned":false,"hidden":false,"mode":"md"},{"id":92,"value":"function findPeaks(audioData, offset, higher) {\n const a = Array();\n let y = audioData[0];\n let next = audioData[1];\n if (higher(y, next)) {\n a.push({ t: offset, y });\n }\n let prev;\n for (let i = 1; i < audioData.length - 1; i++) {\n prev = y;\n y = next;\n next = audioData[i + 1];\n if (higher(y, prev) && higher(y, next)) {\n a.push({ t: offset + i, y });\n }\n }\n if (higher(next, y)) {\n a.push({ t: offset + audioData.length - 1, y: next });\n }\n return a;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":49,"value":"localMaxes = findPeaks(audioData, 0, (a, b) => a > b)","pinned":false,"hidden":false,"mode":"ojs"},{"id":60,"value":"localMins = findPeaks(audioData, 0, (a, b) => a < b)","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"## Filtering high-frequency data for plotting\n\nThis is much less data, but it's still a lot to plot. If zoomed out all the way, it's far more points than the resolution of the screen (horizontally).\n\nThe extremes should always be plotted because they define peak levels. When amplifying some audio, they are the points where the audio will start to be clipped, causing noise. But many local maximums and minimums will be overshadowed by nearby higher peaks and can be removed.\n\nFirst, we need to decide what \"nearby\" means. To do this I calculate roughly how many points there will be per pixel, under conservative assumptions.","pinned":false,"hidden":false,"mode":"md"},{"id":70,"value":"function chooseNearby(sampleLength, plotWidth) {\n return Math.ceil(0.5 * (sampleLength / plotWidth));\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":121,"value":"nearby = chooseNearby(audioData.length, 0.5 * width)","pinned":true,"hidden":false,"mode":"ojs"},{"id":72,"value":"if we assume our plot uses half the width of the notebook and we are plotting the entire recording, *nearby* will mean within ${nearby} samples. We can remove peaks that have a higher peak within that distance with little effect on how the envelope is plotted.\n\n(A subtlety is that the higher peak must be one that we *kept*. For a long sequence of declining peaks, we still want to keep enough points so that each horizontal pixel will have at least one peak.)","pinned":false,"hidden":false,"mode":"md"},{"id":111,"value":"function filterPeaks(peaks, nearby, higher) {\n let kept = null;\n let candidate = null;\n const a = Array();\n for (let d of peaks) {\n if (!candidate) {\n candidate = d;\n if (!kept) {\n a.push(candidate);\n kept = candidate;\n candidate = null;\n }\n continue;\n }\n // invariant: kept and candidate are non-null and different points.\n if (d.t - kept.t > nearby) {\n // no more nearby candidates to check\n a.push(candidate);\n kept = candidate;\n candidate = d;\n } else if (higher(d.y, candidate.y)) {\n candidate = d;\n }\n }\n return a;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":117,"value":"filteredLocalMaxes = filterPeaks(localMaxes, nearby, (a, b) => a > b)","pinned":false,"hidden":false,"mode":"ojs"},{"id":133,"value":"filteredLocalMins = filterPeaks(localMins, nearby, (a, b) => a < b)","pinned":false,"hidden":false,"mode":"ojs"},{"id":137,"value":"That's much fewer points. Here's plot:","pinned":false,"hidden":false,"mode":"md"},{"id":144,"value":"Plot.plot({\n height: 200,\n width: width,\n marks: [\n Plot.ruleY([0], { stroke: \"#888\" }),\n Plot.lineY(filteredLocalMaxes, { x: \"t\", y: \"y\", stroke: \"#833\" }),\n Plot.lineY(filteredLocalMins, { x: \"t\", y: \"y\", stroke: \"#338\" })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":153,"value":"We can also merge the points into a single line chart, which has the effect of coloring in the middle with zig-zag lines.","pinned":false,"hidden":false,"mode":"md"},{"id":158,"value":"function merge(a, b) {\n const out = Array();\n let i = 0;\n let j = 0;\n while (i < a.length && j < b.length) {\n if (a[i].t < b[j].t) {\n out.push(a[i]);\n i++;\n } else {\n out.push(b[j]);\n j++;\n }\n }\n out.push(...a.slice(i));\n out.push(...b.slice(j));\n return out;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":161,"value":"envelope = merge(filteredLocalMaxes, filteredLocalMins)","pinned":false,"hidden":false,"mode":"ojs"},{"id":164,"value":"Plot.plot({\n height: 200,\n width: width,\n marks: [\n Plot.ruleY([0], { stroke: \"#888\" }),\n Plot.lineY(envelope, { x: \"t\", y: \"y\", stroke: \"#338\" })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":169,"value":"## Plotting low-frequency waveforms\n\nWhen we zoom in far enough, we start to have a different problem. The plotted points are too far apart:","pinned":false,"hidden":false,"mode":"md"},{"id":204,"value":"envStart = merge(localMaxes, localMins).slice(0, 100)","pinned":false,"hidden":false,"mode":"ojs"},{"id":182,"value":"Plot.plot({\n height: 200,\n width: width,\n y: { nice: true },\n marks: [\n Plot.ruleY([0], { stroke: \"#888\" }),\n Plot.lineY(envStart, {\n x: \"t\",\n y: \"y\",\n stroke: \"#338\"\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":188,"value":"To fix this, we can add samples to make sure that the gap between adjacent points is no more than *nearby*. (When *nearby* is 1, this means we keep every point from the original data.)","pinned":false,"hidden":false,"mode":"md"},{"id":199,"value":"function enrich(points, audioData, start, end, nearby) {\n const a = Array();\n let kept = null;\n for (let t = start; t < points[0].t; t += nearby) {\n a.push({ t, y: audioData[t] });\n }\n for (let d of points) {\n if (kept) {\n for (let t = kept.t + nearby; t < d.t; t += nearby) {\n a.push({ t, y: audioData[t] });\n }\n }\n a.push(d);\n kept = d;\n }\n let t = end - 1;\n while (t - kept.t > nearby) {\n t -= nearby;\n }\n while (t < end) {\n a.push({ t, y: audioData[t] });\n t += nearby;\n }\n return a;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":212,"value":"enrichedEnvStart = enrich(\n envStart,\n audioData,\n envStart[0].t,\n envStart.at(-1).t,\n 1\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":210,"value":"Plot.plot({\n height: 200,\n width: width,\n y: { nice: true },\n marks: [\n Plot.ruleY([0], { stroke: \"#888\" }),\n Plot.lineY(enrichedEnvStart, {\n x: \"t\",\n y: \"y\",\n stroke: \"#338\"\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":236,"value":"## Putting it together\n\nHere's a class that, given the starting and ending positions of a region within some audio data, returns the data to plot. (It's a class because pre-calculating all the local maximums and minimums ahead of time is faster.)\n","pinned":false,"hidden":false,"mode":"md"},{"id":275,"value":"class Waveform {\n constructor(audioData) {\n this.audioData = audioData;\n this.maxes = findPeaks(audioData, 0, (a, b) => a > b);\n this.mins = findPeaks(audioData, 0, (a, b) => a < b);\n }\n\n selectedData(start, length) {\n start = Math.floor(start);\n length = Math.floor(length);\n const nearby = chooseNearby(length, 0.5 * width);\n const end = start + length;\n\n const maxes = filterPeaks(\n this.maxes.filter((d) => d.t >= start && d.t < end),\n nearby,\n (a, b) => a > b\n );\n const mins = filterPeaks(\n this.mins.filter((d) => d.t >= start && d.t < end),\n nearby,\n (a, b) => a < b\n );\n\n return enrich(merge(maxes, mins), this.audioData, start, end, nearby);\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":284,"value":"waveform = new Waveform(audioData);","pinned":false,"hidden":false,"mode":"ojs"},{"id":263,"value":"zoomed = waveform.selectedData(\n selectionMiddle - selectionLength / 2,\n selectionLength\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":316,"value":"## Demo","pinned":false,"hidden":false,"mode":"md"},{"id":255,"value":"Plot.plot({\n height: 200,\n width: width,\n y: { domain: [-1, 1] },\n marks: [\n Plot.ruleY([0], { stroke: \"#888\" }),\n Plot.lineY(zoomed, {\n x: \"t\",\n y: \"y\",\n stroke: \"#338\"\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":250,"value":"viewof selectionLength = Inputs.range([200, audioData.length], {\n label: \"Zoom (selection size)\",\n step: 1,\n value: audioData.length,\n transform: Math.log\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":301,"value":"viewof selectionMiddle = Inputs.range([0, audioData.length], {\n label: \"Pan (selection middle)\",\n step: 1,\n value: audioData.length / 2\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":361,"value":"Zooms work well, but panning with a range input moves too fast when you zoom in. I'll tackle that problem some other day.","pinned":false,"hidden":false,"mode":"md"}],"files":"$W31","stdlib":"1","titling":"auto"}