31:T50f,The model's accuracy directly impacts the accuracy of your overall results. If the model's accuracy is between ${modelWorstAccuracy}% and ${modelBestAccuracy}%, then in a worst-case scenario up to ${100 - Math.round(modelAccuracy - marginOfError)}% of results could be incorrect. Using the same example, say your findings across your ${j.formatNumber(dataLength)} entries show the following distribution: - 20% of entries categorized as A - 35% of entries categorized as B - 45% of entries categorized as C Based on the model's accuracy, you can be reasonably confident that the true values for each category fall within these ranges: - ${Math.max(0, Math.round(20 - modelWorstMarginOfError))}% to ${Math.min(100, Math.round(20 + modelWorstMarginOfError))}% for category A - ${Math.max(0, Math.round(35 - modelWorstMarginOfError))}% to ${Math.min(100, Math.round(35 + modelWorstMarginOfError))}% for category B - ${Math.max(0, Math.round(45 - modelWorstMarginOfError))}% to ${Math.min(100, Math.round(45 + modelWorstMarginOfError))}% for category C Note that this is a _worst-case scenario_ approach. If you are comfortable wrangling with data, you can calculate the model's specific margin of error for each individual category and use those instead of applying a flat rate across the board.32:[] 2f:{"title":"How many entries should I double-check when using AI?","cells":[{"id":0,"value":"# How many entries should I double-check when using AI?","pinned":false,"hidden":false,"mode":"md"},{"id":3,"value":"When you use AI models for data extraction, classification, cleaning and more, you should double-check the results because the models are a \"black box.\" Unlike traditional code, you can't be sure what they are doing.\n\nHowever, you can use standard statistical methods to estimate how many entries you need to review to be confident the results fall within a chosen margin of error.\n\nUse the calculator below to estimate how many entries you need to manually double-check to have confidence in the results the model provides.","pinned":false,"hidden":false,"mode":"md"},{"id":35,"value":"viewof dataLength = Inputs.range([10, 1_000_000], {\n label: \"Number of entries in your dataset\",\n step: 1,\n value: 1000\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"viewof confidenceLevel = Inputs.radio([90, 95, 99], {\n label: \"Confidence (%)\",\n value: 95\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":28,"value":"viewof marginOfError = Inputs.range([1, 15], {\n label: \"Margin of error (%)\",\n step: 1,\n value: 5\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"

With ${j.formatNumber(dataLength)} entries processed by AI, you need to manually double-check ${sampleToDoubleCheck} random results to be ${confidenceLevel}% confident in the accuracy of the results, allowing for a ${marginOfError}% margin of error.

","pinned":false,"hidden":false,"mode":"html"},{"id":187,"value":"Once you have double-checked the recommended sample, you can use the number of errors the model made to estimate its overall accuracy.","pinned":false,"hidden":false,"mode":"md"},{"id":189,"value":"viewof dataErrors = Inputs.range([0, sampleToDoubleCheck], {\n label: \"Number of errors made by the AI in the random sample\",\n step: 1,\n value: Math.round(sampleToDoubleCheck * 0.05)\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"

If the model made ${j.formatNumber(dataErrors)} errors in a random sample of ${j.formatNumber(sampleToDoubleCheck)} results, its accuracy across the entire dataset of ${j.formatNumber(dataLength)} entries is likely between ${modelWorstAccuracy}% and ${modelBestAccuracy}%. This means the model could be wrong up to ${modelWorstMarginOfError}% of the time.

","pinned":false,"hidden":false,"mode":"html"},{"id":98,"value":"## What does this mean?","pinned":false,"hidden":false,"mode":"md"},{"id":153,"value":"### Model accuracy","pinned":false,"hidden":false,"mode":"md"},{"id":257,"value":"For example, say you classified your ${j.formatNumber(dataLength)} entries into categories A, B, and C using AI. After double-checking a random sample of ${j.formatNumber(sampleToDoubleCheck)} results, you discovered the AI model was wrong for ${j.formatNumber(dataErrors, {decimals: 0})} entries (${Math.min(100, Math.round(dataErrors / sampleToDoubleCheck * 100))}%).\n\nThat means your model was correct ${modelAccuracy}% of the time in this specific random sample. Because your margin of error is ${marginOfError}%, you can be reasonably confident that the model's true accuracy across the entire dataset is between ${modelWorstAccuracy}% and ${modelBestAccuracy}%.\n\nA ${confidenceLevel}% confidence level means that if you were to select a new random sample of ${j.formatNumber(sampleToDoubleCheck)} entries 100 times, you would expect the true model accuracy to fall within this range ${confidenceLevel} times.","pinned":false,"hidden":false,"mode":"md"},{"id":177,"value":"### Results accuracy","pinned":false,"hidden":false,"mode":"md"},{"id":261,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":163,"value":"### And so what?","pinned":false,"hidden":false,"mode":"md"},{"id":264,"value":"Now that you have an estimate of your results' accuracy, you need to decide whether the margin is acceptable for your analysis.\n\nIf it isn't, you can refine your prompt, choose a different AI model or process the data manually.","pinned":false,"hidden":false,"mode":"md"},{"id":118,"value":"## Other things that might interest you","pinned":false,"hidden":false,"mode":"md"},{"id":120,"value":"The code performing the calculation is from the [open-source journalism library](https://github.com/nshiab/journalism), specifically the [getSampleSizeProportion function](https://github.com/nshiab/journalism-statistics/blob/main/src/statistics/getSampleSizeProportion.ts).\n\nCheck out [Maths for journalists](https://observablehq.com/@nshiab/math-for-journalists) for more mathematical concepts.\n\nFollow [Code Like a Journalist](https://www.code-like-a-journalist.com/) if you want to learn how to use the [simple-data-analysis](https://github.com/nshiab/simple-data-analysis), among other things.","pinned":false,"hidden":false,"mode":"md"},{"id":22,"value":"____\n\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":18,"value":"j = import(\"https://esm.sh/jsr/@nshiab/journalism@2.5.2/web\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":59,"value":"sampleToDoubleCheck = j.getSampleSizeProportion(\n dataLength,\n confidenceLevel,\n marginOfError\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"modelAccuracy = 100 -\n Math.min(Math.round((dataErrors / sampleToDoubleCheck) * 100), 100)","pinned":false,"hidden":false,"mode":"ojs"},{"id":229,"value":"modelWorstAccuracy = Math.max(0, modelAccuracy - marginOfError)","pinned":false,"hidden":false,"mode":"ojs"},{"id":232,"value":"modelBestAccuracy = Math.min(100, modelAccuracy + 5)","pinned":false,"hidden":false,"mode":"ojs"},{"id":250,"value":"modelWorstMarginOfError = 100 - modelWorstAccuracy","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}