30:T4ad,md`# Plateau detection How can we identify plateaus and inflexion points in a dataset? The aim is to separate the positive regimes (sections where the data increases—or, alternatively, increases faster than not), the plateaus (where the data doesn't change much, or is close to its median regime), and negative regimes. Googling the question leads to lots of folk recipes, so… I'll implement my own recipe. (If you are a real statistician, please pardon me and send suggestions.) The recipe in this notebook: Interpolate the points to a curve with a [loess regression](https://observablehq.com/@harrystevens/loess-regression). (An immediate advantage of this method, on top of the noise suppression, is that we retrieve a limited number of equally-spaced values.) Use a difference operator to infer its variation. Where the variation is higher than 0 (or its median), we're in a green zone; when it's lower, it's the red zone. (In addition, we filter out the smallest zones.) Inflexion points are the threshold values that separate the zones. To identify plateaus (in light blue color), look for regions where the regime is close to 0 (or its median), within a certain tolerance.`31:T6bb,chart = { const svg = d3.select( DOM.svg( innerWidth + margin.left + margin.right, innerHeight + margin.top + margin.bottom ) ); const g = svg .append("g") .attr("transform", `translate(${margin.left}, ${margin.top})`); const color = show_plateaus ? ["rgb(0,170,232)" /* lightblue */, "white"] : ["lime", "red"]; g.selectAll("rect") .data(d3.pairs(thresholds)) .join("rect") .attr("fill", d => color[+(d[0][1] < 0)]) .style("opacity", 0.1) .attr("x", d => xScale(d[0][0])) .attr("width", d => xScale(d[1][0]) - xScale(d[0][0])) .attr("height", innerHeight); g.append("g") .attr("class", "axis") .call(xAxisGenerator); g.append("g") .attr("class", "axis") .attr("transform", `translate(${innerWidth})`) .call(yAxisGenerator); g.selectAll("circle") .data(data) .enter() .append("circle") .attr("r", 2) .attr("cx", d => xScale(d.year)) .attr("cy", d => yScale(d.value)); g.append("path") .attr("class", "regression") .datum(r) .attr("d", lineGenerator.defined(true)); if (false) { g.append("path") .attr("class", "regression") .style("stroke", "lime") .datum(r) .attr("d", lineGenerator.defined((d, i) => diffs[i] > 0)); g.append("path") .attr("class", "regression") .style("stroke", "red") .datum(r) .attr("d", lineGenerator.defined((d, i) => diffs[i] < 0)); } g.selectAll(".inflexion") .data(thresholds.slice(1, -1)) .join("rect") .attr("fill", "white") .attr("stroke", "black") .attr("x", d => Math.round(xScale(d[0])) - 1) .attr("width", 2) .attr("height", 2); return svg.node(); }32:T19cd,data0 = [{"state_name":"California","year":1901,"value":57.3214},{"state_name":"California","year":1902,"value":56.2134},{"state_name":"California","year":1903,"value":56.317},{"state_name":"California","year":1904,"value":57.4467},{"state_name":"California","year":1905,"value":57.1332},{"state_name":"California","year":1906,"value":57.2392},{"state_name":"California","year":1907,"value":56.5545},{"state_name":"California","year":1908,"value":56.2721},{"state_name":"California","year":1909,"value":56.2912},{"state_name":"California","year":1910,"value":57.7532},{"state_name":"California","year":1911,"value":55.474},{"state_name":"California","year":1912,"value":55.5628},{"state_name":"California","year":1913,"value":56.5263},{"state_name":"California","year":1914,"value":57.4455},{"state_name":"California","year":1900,"value":57.5605},{"state_name":"California","year":1915,"value":56.8189},{"state_name":"California","year":1916,"value":55.5932},{"state_name":"California","year":1917,"value":56.6997},{"state_name":"California","year":1918,"value":56.9079},{"state_name":"California","year":1919,"value":56.3611},{"state_name":"California","year":1920,"value":56.1257},{"state_name":"California","year":1921,"value":57.3778},{"state_name":"California","year":1922,"value":56.1452},{"state_name":"California","year":1923,"value":56.4466},{"state_name":"California","year":1924,"value":57.1546},{"state_name":"California","year":1925,"value":57.206},{"state_name":"California","year":1926,"value":58.8471},{"state_name":"California","year":1927,"value":56.94},{"state_name":"California","year":1928,"value":57.7148},{"state_name":"California","year":1929,"value":57.3422},{"state_name":"California","year":1930,"value":56.9745},{"state_name":"California","year":1931,"value":58.3638},{"state_name":"California","year":1932,"value":56.9607},{"state_name":"California","year":1933,"value":56.8247},{"state_name":"California","year":1934,"value":59.7805},{"state_name":"California","year":1935,"value":56.8945},{"state_name":"California","year":1936,"value":58.6637},{"state_name":"California","year":1937,"value":57.2449},{"state_name":"California","year":1938,"value":57.3329},{"state_name":"California","year":1939,"value":58.4247},{"state_name":"California","year":1940,"value":58.8798},{"state_name":"California","year":1941,"value":57.1222},{"state_name":"California","year":1942,"value":57.1411},{"state_name":"California","year":1943,"value":57.894},{"state_name":"California","year":1944,"value":56.2842},{"state_name":"California","year":1945,"value":57.1573},{"state_name":"California","year":1946,"value":56.8022},{"state_name":"California","year":1947,"value":57.6003},{"state_name":"California","year":1948,"value":55.6891},{"state_name":"California","year":1949,"value":56.2792},{"state_name":"California","year":1950,"value":58.1899},{"state_name":"California","year":1951,"value":57.2899},{"state_name":"California","year":1952,"value":56.8036},{"state_name":"California","year":1953,"value":57.2433},{"state_name":"California","year":1954,"value":57.6142},{"state_name":"California","year":1955,"value":56.3011},{"state_name":"California","year":1956,"value":57.0172},{"state_name":"California","year":1957,"value":57.1288},{"state_name":"California","year":1958,"value":58.9603},{"state_name":"California","year":1959,"value":59.0416},{"state_name":"California","year":1960,"value":58.0866},{"state_name":"California","year":1961,"value":57.8652},{"state_name":"California","year":1962,"value":57.4384},{"state_name":"California","year":1963,"value":57.1077},{"state_name":"California","year":1964,"value":56.6202},{"state_name":"California","year":1965,"value":56.774},{"state_name":"California","year":1966,"value":58.0992},{"state_name":"California","year":1967,"value":57.5668},{"state_name":"California","year":1968,"value":57.5989},{"state_name":"California","year":1969,"value":57.5899},{"state_name":"California","year":1970,"value":57.8767},{"state_name":"California","year":1971,"value":56.2718},{"state_name":"California","year":1972,"value":57.2934},{"state_name":"California","year":1973,"value":57.3641},{"state_name":"California","year":1974,"value":57.7036},{"state_name":"California","year":1975,"value":56.3447},{"state_name":"California","year":1976,"value":57.5107},{"state_name":"California","year":1977,"value":58.0088},{"state_name":"California","year":1978,"value":57.6893},{"state_name":"California","year":1979,"value":57.7485},{"state_name":"California","year":1980,"value":58.1052},{"state_name":"California","year":1981,"value":59.3551},{"state_name":"California","year":1982,"value":56.4003},{"state_name":"California","year":1983,"value":57.6184},{"state_name":"California","year":1984,"value":58.2609},{"state_name":"California","year":1985,"value":57.3929},{"state_name":"California","year":1986,"value":58.9427},{"state_name":"California","year":1987,"value":58.3293},{"state_name":"California","year":1988,"value":58.7852},{"state_name":"California","year":1989,"value":58.1885},{"state_name":"California","year":1990,"value":58.0488},{"state_name":"California","year":1991,"value":58.16},{"state_name":"California","year":1992,"value":59.2918},{"state_name":"California","year":1993,"value":57.7408},{"state_name":"California","year":1994,"value":58.1055},{"state_name":"California","year":1995,"value":58.9677},{"state_name":"California","year":1996,"value":59.594},{"state_name":"California","year":1997,"value":59.1923},{"state_name":"California","year":1998,"value":56.7942},{"state_name":"California","year":1999,"value":58.0356},{"state_name":"California","year":2000,"value":58.8363},{"state_name":"California","year":2001,"value":59.2216},{"state_name":"California","year":2002,"value":58.8964},{"state_name":"California","year":2003,"value":59.529},{"state_name":"California","year":2004,"value":58.9128},{"state_name":"California","year":2005,"value":58.6833},{"state_name":"California","year":2006,"value":58.6975},{"state_name":"California","year":2007,"value":58.9929},{"state_name":"California","year":2008,"value":58.9459},{"state_name":"California","year":2009,"value":58.917},{"state_name":"California","year":2010,"value":57.7926},{"state_name":"California","year":2011,"value":57.5186},{"state_name":"California","year":2012,"value":59.5448},{"state_name":"California","year":2013,"value":59.3921},{"state_name":"California","year":2014,"value":61.5208},{"state_name":"California","year":2015,"value":60.7992},{"state_name":"California","year":2016,"value":60.1429},{"state_name":"California","year":2017,"value":60.4208}]33:[] 2e:{"title":"Plateau detection","cells":[{"id":0,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":278,"value":"viewof bandwidth = slider({\n title: \"LOESS Bandwidth\",\n min: .02,\n max: 1,\n step: .01,\n value: .25\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":520,"value":"viewof compare_to = radio({\n options: [\n { label: \"the median of the differences\", value: \"median\" },\n \"zero\"\n ],\n value: \"median\",\n title: \"Compare the regime to:\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":148,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":516,"value":"viewof show_plateaus = checkbox({\n options: [\"show plateaus\"],\n value: [\"show plateaus\"]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":572,"value":"viewof plateau_tolerance = slider({\n title: \"Plateau tolerance\",\n min: 0,\n max: 1.5,\n step: 0.01,\n value: 0.5,\n format: \".0%\",\n description: \"% of the standard deviation of the differences\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":654,"value":"viewof dataSource = select({\n options: [\n \"original example\",\n \"squared\",\n \"sinusoid\",\n \"log\",\n \"staircase\",\n \"combined\"\n ],\n description: \"Data source\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":417,"value":"r = regressionGenerator(data)","pinned":true,"hidden":false,"mode":"ojs"},{"id":419,"value":"function difference(data, k = 1) {\n const sob = [];\n for (let i = 0; i < data.length; i++) {\n const j = i < k ? k : i + k >= data.length ? data.length - 1 - k : i;\n sob[i] = 0;\n for (let u = 1; u <= k; u++)\n sob[i] += (data[j + u][1] - data[j - u][1]) / k;\n }\n return sob;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":421,"value":"diffs = {\n const s = difference(r),\n m = compare_to.match(/median/) ? d3.median(s) : 0; // median is more robust to outliers than mean\n return s.map(d => d - m || 1e-12) /* enforce +sign for the median */;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":546,"value":"plateaus = {\n let i = 0;\n const t = [],\n stddev = d3.deviation(diffs),\n zeros = diffs.map(d => (Math.abs(d) < plateau_tolerance * stddev ? 1 : -1));\n for (const [a, b] of d3.pairs(zeros)) {\n if (a * b < 0 || i == 0 || i == diffs.length - 2)\n t.push([r[i][0], Math.sign(b)]);\n i++;\n }\n return t;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":465,"value":"regimes = {\n const t = [];\n let i = 0;\n for (const [a, b] of d3.pairs(diffs)) {\n if (a * b < 0 || i == 0 || i == diffs.length - 2)\n t.push([r[i][0], Math.sign(b)]);\n i++;\n }\n return t;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":487,"value":"// filter out small slivers\nthresholds = {\n const t = (show_plateaus ? plateaus : regimes).slice();\n for (let i = 1; i < t.length - 1; i++) {\n if (t[i + 1][0] - t[i][0] < 4) {\n t[i] = t[i + 1] = null;\n i++;\n }\n }\n return t.filter(d => d);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":616,"value":"data = {\n switch (dataSource) {\n case \"squared\":\n return d3.range(1901, 2020, 1).map(year => ({\n year,\n value: ((year - 1930) / 10) ** 2\n }));\n case \"sinusoid\":\n return d3.range(1901, 2020, 0.2).map(year => ({\n year,\n value: 10 + 5 * Math.sin(year / 5)\n }));\n case \"log\":\n return d3.range(1901, 2020, 2).map(year => ({\n year,\n value: 2 * Math.log(year - 1900) + .5 * Math.random()\n }));\n case \"staircase\":\n return d3.range(1901, 2020, 3).map(year => ({\n year,\n value: (year / 30) | 0\n }));\n case \"combined\":\n return d3.range(1901, 2020, 0.2).map(year => ({\n year,\n value: year / 5 + 5 * Math.sin(year / 5) + Math.random()\n }));\n default:\n return data0;\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":500,"value":"md`_Project developed for the [LIRIS M2i project](https://projet.liris.cnrs.fr/mi2/) by Philippe Rivière._\n\n----\n_The part below this line is Harry Stevens’ original notebook._`","pinned":false,"hidden":false,"mode":"ojs"},{"id":292,"value":"md`[d3-regression](https://github.com/HarryStevens/d3-regression)'s implementation of the LOESS regression was adapted with very little modification from Jason Davies's [science.js](https://github.com/jasondavies/science.js), with the main difference that this API returns a configurable function with x- and y-accessors similar to other D3.js modules.\n\nUse d3.[regressionLoess()](https://github.com/HarryStevens/d3-regression#regressionQuadratic) like this:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":34,"value":"regressionGenerator = d3.regressionLoess()\n .x(d => d.year)\n .y(d => d.value)\n .bandwidth(bandwidth);","pinned":true,"hidden":false,"mode":"ojs"},{"id":360,"value":"md`You can use d3.[line()](https://github.com/d3/d3-shape/#line) to draw the line to the page:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":356,"value":"lineGenerator = d3.line()\n .x(d => xScale(d[0]))\n .y(d => yScale(d[1]));","pinned":true,"hidden":false,"mode":"ojs"},{"id":26,"value":"md`**d3-regression** is a [D3.js](https://d3js.org/) module for calculating statistical regressions from two-dimensional data. It is dependency-free. In addition to LOESS regressions, it can also do [linear](https://observablehq.com/d/a2ccc2b456a3c927), [quadratic](https://observablehq.com/d/1579404ed3ac2fe5), [logarithmic](https://observablehq.com/d/9d575ab9baa6a001), [exponential](https://observablehq.com/d/53a55eb2fe6dc02c), and [power law](https://observablehq.com/@harrystevens/power-law-regression) regressions.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":114,"value":"md`## Data`","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":150,"value":"md`## Dimensions`","pinned":false,"hidden":false,"mode":"ojs"},{"id":152,"value":"margin = ({ left: 44, right: 11, top: 14, bottom: 21 })","pinned":false,"hidden":false,"mode":"ojs"},{"id":153,"value":"base = Math.min(width, 400)","pinned":false,"hidden":false,"mode":"ojs"},{"id":156,"value":"innerWidth = base - margin.left - margin.right","pinned":false,"hidden":false,"mode":"ojs"},{"id":158,"value":"innerHeight = base - margin.top - margin.bottom","pinned":false,"hidden":false,"mode":"ojs"},{"id":136,"value":"md`## Scales`","pinned":false,"hidden":false,"mode":"ojs"},{"id":144,"value":"xScale = d3.scaleLinear()\n .domain([1900, 2020])\n .range([0, innerWidth])","pinned":false,"hidden":false,"mode":"ojs"},{"id":145,"value":"yScale = d3\n .scaleLinear()\n .domain(d3.extent(data, d => d.value))\n .range([innerHeight, 0])\n .nice()","pinned":false,"hidden":false,"mode":"ojs"},{"id":130,"value":"md`## Axes`","pinned":false,"hidden":false,"mode":"ojs"},{"id":131,"value":"xAxisGenerator = d3.axisBottom(xScale)\n .tickSize(innerHeight + 10)\n .tickValues(d3.range(1900, 2040, 20))\n .tickFormat(d => d % 100 === 0 ? d : \"`\" + d.toString().slice(2, 4))","pinned":false,"hidden":false,"mode":"ojs"},{"id":133,"value":"yAxisGenerator = d3\n .axisLeft(yScale)\n .tickSize(innerWidth + 10)\n .tickFormat(d => d)","pinned":false,"hidden":false,"mode":"ojs"},{"id":119,"value":"md`## Styles`","pinned":false,"hidden":false,"mode":"ojs"},{"id":127,"value":"html`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":139,"value":"md`## Imports\n\nTo make the scatterplot, I used [d3-axis](https://github.com/d3/d3-axis), [d3-scale](https://github.com/d3/d3-scale), and [d3-selection](https://github.com/d3/d3-selection). To draw the regression line between points, I used [d3-shape](https://github.com/d3/d3-selection). I also am loading [d3-regression](https://github.com/HarryStevens/d3-regression) with the rest of the D3.js modules.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":142,"value":"d3 = require.alias({\n \"d3-array\": \"d3-array@2\",\n \"d3-axis\": \"d3-axis@1\",\n \"d3-regression\": \"d3-regression@1\",\n \"d3-scale\": \"d3-scale@3\",\n \"d3-shape\": \"d3-shape@1\",\n \"d3-selection\": \"d3-selection@1\"\n})(\"d3-array\", \"d3-axis\", \"d3-regression\", \"d3-scale\", \"d3-shape\", \"d3-selection\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":279,"value":"import { checkbox, radio, select, slider } from '@jashkenas/inputs'","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}