31:T50b,{ const { corpus, counts: c, calibDocs } = model; const single = Math.max(corpus - calibDocs, 0); const segs = [ { k: "6 reviewers", n: c[6], col: "#085041" }, { k: "5 reviewers", n: c[5], col: "#0F6E56" }, { k: "4 reviewers", n: c[4], col: "#1D9E75" }, { k: "3 reviewers", n: c[3], col: "#5DCAA5" }, { k: "2 reviewers", n: c[2], col: "#9FE1CB" }, { k: "1 reviewer", n: c[1], col: "#D3D1C7" }, { k: "LLM review", n: single, col: "#ECEAE3" } ].filter((s) => s.n > 0); const pct = (n) => ((100 * n) / corpus).toFixed(1); return html`

${segs.map( (s) => html`

` )}

${segs.map( (s) => html`${s.k} · ${pct(s.n)}%` )}

`; } 32:T90b,model = { const z = 1.96; const { n1, n2, n3, n4, n5, n6 } = design; const { rho, aiHum, aiConc, aiRuns } = measured; const { Rtarget, ciTarget } = targets; const { minutesPerReview, hoursPerCoderDay } = effort; const counts = { 1: n1, 2: n2, 3: n3, 4: n4, 5: n5, 6: n6 }; const calibDocs = [1, 2, 3, 4, 5, 6].reduce((s, k) => s + counts[k], 0); const reviews = [1, 2, 3, 4, 5, 6].reduce((s, k) => s + counts[k] * k, 0); const wAt = (r) => { let info = 0; for (const k of [1, 2, 3, 4, 5, 6]) info += (counts[k] * (k * (k - 1))) / (1 + (k - 1) * r) ** 2; return info > 0 ? Math.sqrt((8 * z * z * (1 - r) ** 2) / info) : Infinity; }; const ciWidth = wAt(rho); let breakeven = null; for (let r = 0.4; r <= 0.99; r += 0.005) { if (wAt(r) <= ciTarget) { breakeven = r; break; } } const aiValRaw = (aiHum * aiHum) / rho; const correlated = aiValRaw >= 1; const rA = Math.min(0.98, aiValRaw); const m = aiRuns, cEff = Math.max(aiConc, rA + 0.001); const ceiling = Math.min(0.98, rA / cEff); const effAI = Math.min(ceiling, (rA * m) / (1 + (m - 1) * cEff)); const eH = (1 - rho) / rho, eAI = (1 - effAI) / effAI; const Rwith = (h) => 1 / (1 + (h * eH + eAI) / (h + 1) ** 2); const Rhum = (k) => (k > 0 ? (k * rho) / (1 + (k - 1) * rho) : 0); let hAI = null; for (let h = 0; h <= 6; h++) if (Rwith(h) >= Rtarget) { hAI = h; break; } let kHum = null; for (let k = 1; k <= 6; k++) if (Rhum(k) >= Rtarget) { kHum = k; break; } const capped = hAI === null, hPasses = capped ? 6 : hAI, kHumOnly = kHum === null ? 6 : kHum; const prodDocs = Math.max(corpus - calibDocs, 0); const daysFor = (p) => Math.ceil( (((prodDocs * p) / 6) * minutesPerReview) / 60 / hoursPerCoderDay ); const prodReviews = prodDocs * hPasses, prodPerCoder = prodReviews / 6; const prodDays = daysFor(hPasses), baseDays = daysFor(kHumOnly); return { corpus, counts, calibDocs, reviews, ciWidth, breakeven, ciTarget, rho, Rtarget, aiHum, aiConc, aiRuns, rA, effAI, ceiling, correlated, hPasses, capped, kHumOnly, prodDocs, prodReviews, prodPerCoder, prodDays, baseDays }; }33:T6aa,{ const { breakeven, ciTarget, ciWidth, rho } = model; const lo = 0.4, hi = 0.99, axW = 300, xp = (r) => ((r - lo) / (hi - lo)) * axW, beX = breakeven !== null ? xp(breakeven) : 0; return html`

At ρ = ${rho}, this design gives a calibration interval of ±${( ciWidth / 2 ).toFixed(3)}.

${ breakeven !== null ? html`

It meets your ±${(ciTarget / 2).toFixed( 3 )} target as long as true reliability is at least ρ = ${breakeven.toFixed( 2 )}.

` : html`

It never reaches your ±${(ciTarget / 2).toFixed( 3 )} target — widen the target or add documents.

` }

${ breakeven !== null ? html`

` : "" } ${ breakeven !== null ? html`

` : "" }

red = misses · green = meets · blue line = current ρ

`; }34:T9da,--- ## Further reading **[Inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability) / [intraclass correlation](https://en.wikipedia.org/wiki/Intraclass_correlation) — the ρ slider** - Shrout, P. E. & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. *Psychological Bulletin*, 86(2), 420–428. - McGraw, K. O. & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. *Psychological Methods*, 1(1), 30–46. - Categorical codes rather than scores: [Cohen's κ](https://en.wikipedia.org/wiki/Cohen%27s_kappa), [Fleiss' κ](https://en.wikipedia.org/wiki/Fleiss%27_kappa) for many raters, and [Krippendorff's α](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) for ordinal data with missing ratings. - Bonett, D. G. (2002). Sample size requirements for estimating intraclass correlations with desired precision. *Statistics in Medicine*, 21(9), 1331–1335. — [doi:10.1002/sim.1108](https://doi.org/10.1002/sim.1108) **[Combining several reviewers into one verdict](https://en.wikipedia.org/wiki/Spearman%E2%80%93Brown_prediction_formula) — passes needed** - The Spearman–Brown prophecy formula: Spearman, C. (1910) and Brown, W. (1910), *British Journal of Psychology*, 3. Covered in any psychometrics text. - [Generalizability theory](https://en.wikipedia.org/wiki/Generalizability_theory), for designs with varying numbers of raters per item: Cronbach, Gleser, Nanda & Rajaratnam (1972), *The Dependability of Behavioral Measurements*; Brennan, R. L. (2001), *Generalizability Theory*. **[Recovering AI accuracy from AI–human concordance](https://en.wikipedia.org/wiki/Correction_for_attenuation) — the disattenuation step** - Correction for attenuation: Spearman, C. (1904). The proof and measurement of association between two things. *American Journal of Psychology*, 15, 72–101. - Lord, F. M. & Novick, M. R. (1968). *Statistical Theories of Mental Test Scores* — the foundational treatment of true scores, reliability, and error. **Using AI as a rater** - Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. *NeurIPS*. — [arXiv:2306.05685](https://arxiv.org/abs/2306.05685) - Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement (ICLR 2025) — [paper](https://proceedings.iclr.cc/paper_files/paper/2025/file/08dabd5345b37fffcbe335bd578b15a0-Paper-Conference.pdf). Trust the model only when it's confident humans would agree, and escalate otherwise. 35:[] 2f:{"title":"Accreditation Review Protocol","cells":[{"id":0,"value":"# Accreditation Review Protocol\nInteract with the sliders and inputs below to explore options for our HLC evidence review protocol.","pinned":false,"hidden":false,"mode":"md"},{"id":66,"value":"---\n## 1 · Calibration design\n\nThe slider below set how many documents we will review at each reviewer \"depth\". The \"6 reviewers\" slider sets how many documents are reviewed independently by six coders; The \"1 reviewer\" slider sets how many documents are reviewed by just one. Deeper tiers measure agreement precisely; shallow tiers add breadth across more documents. Single-reviewer documents add coverage only — a lone reviewer has no one to agree or disagree with, with no impact on concordance or reliability.","pinned":false,"hidden":false,"mode":"md"},{"id":15,"value":"viewof corpus = Inputs.range([0, 1000], {\n value: 800,\n label: \"Evidence Document Count\",\n step: 1\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":78,"value":"viewof design = Inputs.form({\n n6: Inputs.range([0, 80], {\n value: 10,\n step: 1,\n label: \"Docs · 6 reviewers\"\n }),\n n5: Inputs.range([0, 80], {\n value: 5,\n step: 1,\n label: \"Docs · 5 reviewers\"\n }),\n n4: Inputs.range([0, 80], { value: 3, step: 1, label: \"Docs · 4 reviewers\" }),\n n3: Inputs.range([0, 80], {\n value: 5,\n step: 1,\n label: \"Docs · 3 reviewers\"\n }),\n n2: Inputs.range([0, 80], {\n value: 15,\n step: 1,\n label: \"Docs · 2 reviewers\"\n }),\n n1: Inputs.range([0, 80], { value: 0, step: 1, label: \"Docs · 1 reviewer\" })\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":81,"value":"{\n const color = {\n 1: \"#D3D1C7\",\n 2: \"#9FE1CB\",\n 3: \"#5DCAA5\",\n 4: \"#1D9E75\",\n 5: \"#0F6E56\",\n 6: \"#085041\"\n };\n const c = model.counts;\n return html`

${[\n 6, 5, 4, 3, 2, 1\n ].map(\n (\n k\n ) => html`

\n ${k} reviewer${\n k === 1 ? \"\" : \"s\"\n }\n

\n ${\n c[k] ? c[k] + \" docs\" : \"\"\n }\n

`\n )}

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"The bar below shows what slice of all 800 documents each depth covers; \"LLM review\" is everything not in the calibration set, which gets handled in the production phases below.","pinned":false,"hidden":false,"mode":"md"},{"id":88,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":271,"value":"- **${model.calibDocs}** documents are reviewed.\n- **${ Math.ceil(Object.entries(model.counts)\n .map((d) => parseInt(d[0]) * parseInt(d[1]))\n .reduce((a, v) => (a += v))) }** reviews are conducted in total.\n- Each reviewer reviews **${ Math.ceil(Object.entries(model.counts)\n .map((d) => parseInt(d[0]) * parseInt(d[1]))\n .reduce((a, v) => (a += v)) /6) }** documents.","pinned":false,"hidden":false,"mode":"md"},{"id":92,"value":"---\n## 2 · Measurements\n\nThe Four numbers below would be calculated on the initial calibration cycle above. But also, play with them to simulate potential outcomes!\n\n- **Human reliability ρ**: how consistently our coders agree with one another, pooled across *every* document that got two or more reviewers. A 6-reviewer doc carries more weight than a 2-reviewer one. Lower values indicate that documents are harder to review or fundamentally more variable, so more review is needed everywhere downstream.\n- **LLM–human concordance**: how often and how closely an LLM score matches a human score. \n- **LLM concordance**: how consistently the LLM agrees with itself across repeat runs, or how closely different LLMs score documents.\n- **LLM reviews per doc** — how many times each document is analyzed by the same or distinct, non-deterministic LLMs. ","pinned":false,"hidden":false,"mode":"md"},{"id":96,"value":"viewof measured = Inputs.form({\n rho: Inputs.range([0.4, 0.95], {\n value: 0.8,\n step: 0.01,\n label: \"Human reliability ρ\"\n }),\n aiHum: Inputs.range([0.3, 0.95], {\n value: 0.7,\n step: 0.01,\n label: \"LLM–human concordance\"\n }),\n aiRuns: Inputs.range([1, 5], {\n value: 2,\n step: 1,\n label: \"LLM reviews per doc\"\n }),\n aiConc: Inputs.range([0.5, 0.99], {\n value: 0.85,\n step: 0.01,\n label: \"LLM concordance (run-to-run or model-to-model)\"\n })\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":99,"value":"---\n## 3 · Targets\n\n- **Target per-doc reliability**: how trustworthy each document's final verdict must be. Higher targets demand more reviewers per document.\n- **Target CI width**: how tight the calibration reliability estimate needs to be. This is the bar the calibration design has to clear. Note that this is *full width* CI, not a plus-or-minus value.","pinned":false,"hidden":false,"mode":"md"},{"id":101,"value":"viewof targets = Inputs.form({\n Rtarget: Inputs.range([0.7, 0.98], {\n value: 0.85,\n step: 0.01,\n label: \"Target per-doc reliability\"\n }),\n ciTarget: Inputs.range([0.05, 0.4], {\n value: 0.2,\n step: 0.01,\n label: \"Target CI width\"\n })\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":104,"value":"---\n## 4 · Effort assumptions\n\nUsed only to turn review counts into a projected timeline.","pinned":false,"hidden":false,"mode":"md"},{"id":106,"value":"viewof effort = Inputs.form({\n minutesPerReview: Inputs.range([5, 60], {\n value: 10,\n step: 1,\n label: \"Minutes per review\"\n }),\n hoursPerCoderDay: Inputs.range([1, 8], {\n value: 3,\n step: 0.5,\n label: \"Review-hours per coder/day\"\n })\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":112,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":116,"value":"---\n## Calibration precision\n\nGiven the settings above, how reliable will coding efforts be? The bar runs from low reliability on the left to high on the right. The blue line is the reliability set above; ideally it sits in the green zone, indicating that the review design satisfies the target metrics. If the blue line lands in the red, the design is too thin for the precision sought, and more documents or a looser target are needed. If the blue line is far to the right, human effort is wasted for the given confidence target.","pinned":false,"hidden":false,"mode":"md"},{"id":119,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":123,"value":"---\n## Is the LLM even helping?\n\nAfter correcting for statistical noise and any gain from repeat reviews, is the LLM contributing any reliability? \n\nThe check below compares within-LLM agreement against between LLM-human agreement: if the check below shows red, the LLM is agreeing with coders *too* well, usually a sign it shares their bias rather than correcting it, and the LLM is adding little to human review processes.","pinned":false,"hidden":false,"mode":"md"},{"id":127,"value":"{\n const { aiHum, rA, effAI, ceiling, aiRuns, aiConc, correlated } = model;\n return html`

LLM–human concordance ${aiHum} implies a single-run LLM accuracy of ${rA.toFixed(\n 2\n )} once human noise is divided out.

${aiRuns} run${\n aiRuns === 1 ? \"\" : \"s\"\n } at ${aiConc} self-concordance give an effective LLM reliability of ${effAI.toFixed(\n 2\n )} (ceiling ${ceiling.toFixed(2)}).

\n ${\n correlated\n ? html`

⚠ LLM–human agreement is higher than human reliability allows — the LLM is likely echoing the coders' errors, so treat its contribution with caution.

`\n : html`

Independent-rater check: looks OK at these values.

`\n }\n

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"$34","pinned":false,"hidden":false,"mode":"md"}],"files":"$W35","stdlib":"1","titling":"auto"}