31:T598,# Step 3: Gold standard labeling The display below lists— 1) the variable under dispute, 2) the labels assigned by annotators A and B (both randomly selected), 3) GPT-4o generated explanation and recommendation, and 4) a panel for you to submit your decision. Unless you click the Submit button, your decision won't be recorded. After you click submit, the app will automatically move to the next conflict to resolve. This app is designed to remember your progress. If you accidentally navigate from your browser, close the tab, click refresh, shut down your computer, etc. The browser will still remember your progress. You just upload the same files in step 2, and you should be able to resume from where you left off. If you want to start fresh. If you want the app to forget everything you have done so far, scroll all the way down to the bottom of the notebook titled "Danger zone!!!". Click "Reset all progress". This will allow you to start from scratch. Below you will find a "Refresh" button. Feel free to click it every now and then. It will display the gold standard table as it is in its current state of progress. Note: There is no way to move back on your decision. If you want to retrospectively change your decision, the only way to fix it is to do this manually. Click the "Refresh" button below to identify the row and column of the value you wish to change. Make a note of it and change it at the very end.32:Td30,displayGptAdvice = async ( divID, report, schema, labelA, labelB, { model = "gpt-4o-2024-05-13", temperature = 0.0, seed = 1234 } ) => { const outputElement = document.getElementById(divID); outputElement.textContent = ""; const prompt = ` TASK DESCRIPTION: As an assistant specializing in breast cancer pathology, your role is to impartially understand and help resolve conflicts between two different interpretations of the information contained in a breast cancer pathology report. INSTRUCTIONS: 1. Read the breast cancer pathology report presented under the heading "BREAST CANCER PATHOLOGY REPORT" and enclosed within triple quotes. These reports were digitized using optical character recognition so, expect typographical and sentence alignment errors typical in such documents. 2. Structured JSON data was extracted by two annotators— 'Annotator A' and 'Annotator B'. They were both tasked with extracting the same information from the pathology report under the heading "BREAST CANCER PATHOLOGY REPORT". The specification of the variable, whose value they were both tasked to extract from the report, is presented under the heading "JSON SCHEMA". This section contains the property entry for the variable from the JSON Schema of the extracted dataset. It contains the variable name, description, and constraints specified by the schema. 3. Annotators 'A' and 'B' disagree on the correct value to assign for the variable described under "JSON Schema". Under the heading "LABELS", you are provided both the value determined by Annotator A and Annotator B. 4. Create a heading "EXPLANATION FOR ANNOTATOR A". Read the pathology report under "BREAST CANCER PATHOLOGY REPORT", the JSON Schema under "JSON SCHEMA", and the label assigned by Annotator A under "LABELS". Under the heading "EXPLANATION FOR ANNOTATOR A", write your best impartial explanation for why you think Annotator A assigned the value that they did. Present snippets from the pathology report that supports your claim. If you think the annotator made a mistake, explain the mistake. 5. Create a heading "EXPLANATION FOR ANNOTATOR B". Read the pathology report under "BREAST CANCER PATHOLOGY REPORT", the JSON Schema under "JSON SCHEMA", and the label assigned by Annotator B under "LABELS". Under the heading "EXPLANATION FOR ANNOTATOR B", write your best impartial explanation for why you think Annotator B assigned the value that they did. Present snippets from the pathology report that supports your claim. If you think the annotator made a mistake, explain the mistake. 6. Finally, create a heading "RECOMMENDATION". Under this heading, give your personal opinion on what you think is the correct value to assign for the variable. You are now free to either side with Annotator A, Annotator B, or disagree with them both and choose a different third option. BREAST CANCER PATHOLOGY REPORT: """ ${report} """ JSON SCHEMA: \`\`\`json ${schema} \`\`\` LABELS: - Annotator A label: ${labelA} - Annotator B label: ${labelB} `; const messages = [{ role: "user", content: prompt }]; const stream = await openai.chat.completions.create({ model: model, messages: messages, temperature: temperature, seed: seed, stream: true }); for await (const chunk of stream) { outputElement.textContent += chunk.choices[0]?.delta?.content || ""; } }33:Td6c,async function generateGoldStandardLabels(tableA, tableB) { if (tableA.length !== tableB.length) { throw new Error("Tables do not have the same number of instances."); } if (tableA.length === 0) { throw new Error("Tables should have at least one entry."); } const reports = await getPathologyReports(); const schemas = (await FileAttachment("pie_schemas@1.json").json()) .properties; // Remember progress in the gold standard table let goldStandard = (await localforage.getItem("goldStandard")) || tableA.map((item) => ({ id: item.id })); let lastKeyIndex = (await localforage.getItem("lastKeyIndex")) || 0; let lastRowIndex = (await localforage.getItem("lastRowIndex")) || 0; let keys = Object.keys(tableA[0]).filter((item) => item !== "id"); const toss = () => (Math.random() < 0.5 ? false : true); for (let k = lastKeyIndex; k < keys.length; k++) { // Iterate columns const key = keys[k]; // Catch when a key is not present in both tables if (!tableB[0].hasOwnProperty(key)) { continue; } const schema = `${key}: ${JSON.stringify(schemas[key], null, 2)}`; for ( let i = k === lastKeyIndex ? lastRowIndex : 0; i < tableA.length; i++ ) { // Iterate rows const instanceA = tableA[i]; const instanceB = tableB[i]; if (!goldStandard[i]) { goldStandard[i] = {}; } if ( valuesMatch(instanceA[key], instanceB[key], getType(instanceA[key])) ) { // Annotators agree goldStandard[i][key] = instanceA[key]; } else { // Resolve conflict const type = getTypeFromSchema(schemas[key]); if (toss()) { displayHeading("heading", key, instanceA[key], instanceB[key]); await displayGptAdvice( "chatgpt-output", reports[instanceA.id], schema, instanceA[key], instanceB[key], { model: "gpt-4o-2024-05-13", temperature: 0.0, seed: 1234 } ); } else { displayHeading("heading", key, instanceB[key], instanceA[key]); await displayGptAdvice( "chatgpt-output", reports[instanceA.id], schema, instanceB[key], instanceA[key], { model: "gpt-4o-2024-05-13", temperature: 0.0, seed: 1234 } ); } goldStandard[i][key] = await displayChoices( "decision", type, schemas[key].hasOwnProperty("enum") ? schemas[key].enum : null ); } // Save the current state of goldStandard and the current key/row index to localforage await localforage.setItem("goldStandard", goldStandard); await localforage.setItem("lastKeyIndex", k); await localforage.setItem("lastRowIndex", i + 1); } // Reset the lastRowIndex after a column is fully resolved await localforage.setItem("lastRowIndex", 0); } // Clear the saved indices after completion await localforage.removeItem("lastKeyIndex"); await localforage.removeItem("lastRowIndex"); const outputElement = document.getElementById("chatgpt-output"); outputElement.innerHTML = "

Congratulations! 🥳🏆🥳 You have successfully created the gold standard for your dataset. Please download the JSON output by the cell below that outputs 'goldStandard' variable.

"; return goldStandard; }34:T1116,displayChoices = async (divID, type, options) => { return new Promise((resolve, reject) => { const container = document.getElementById(divID); container.innerHTML = ""; const submitButton = document.createElement("button"); submitButton.textContent = "Submit"; submitButton.type = "button"; if (type === "string") { // String if (options === null) { // Text input field when no options are available const inputField = document.createElement("input"); inputField.type = "text"; inputField.placeholder = "Enter a string"; container.appendChild(inputField); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const value = inputField.value; resolve(value === "" ? null : value); }); } else { // Radio buttons when options are available options.forEach((option) => { const radioContainer = document.createElement("div"); const radioButton = document.createElement("input"); radioButton.type = "radio"; radioButton.name = "choice"; radioButton.value = option === null ? "null" : option; const label = document.createElement("label"); label.textContent = option === null ? "null" : option; radioContainer.appendChild(radioButton); radioContainer.appendChild(label); container.appendChild(radioContainer); }); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const selectedOption = document.querySelector( `input[name="choice"]:checked` ); if (selectedOption) { resolve( selectedOption.value === "null" ? null : selectedOption.value ); } else { alert("Please select an option"); } }); } } else if (type === "boolean") { ["true", "false"].forEach((option) => { const radioContainer = document.createElement("div"); const radioButton = document.createElement("input"); radioButton.type = "radio"; radioButton.name = "choice"; radioButton.value = option; const label = document.createElement("label"); label.textContent = option; radioContainer.appendChild(radioButton); radioContainer.appendChild(label); container.appendChild(radioContainer); }); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const selectedOption = document.querySelector( `input[name="choice"]:checked` ); if (selectedOption) { resolve(selectedOption.value === "true"); } else { alert("Please select an option"); } }); } else if (type === "integer") { const inputField = document.createElement("input"); inputField.type = "number"; inputField.step = "1"; inputField.placeholder = "Enter an integer value"; container.appendChild(inputField); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const value = inputField.value; if (value === "") { resolve(null); } else { resolve(parseInt(value, 10)); } }); } else if (type === "number") { const inputField = document.createElement("input"); inputField.type = "number"; inputField.placeholder = "Enter a number"; container.appendChild(inputField); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const value = inputField.value; if (value === "") { resolve(null); } else { resolve(parseFloat(value)); } }); } else { // Text input field when no options are available and the type is none of the above const inputField = document.createElement("input"); inputField.type = "text"; inputField.placeholder = "Enter a string"; container.appendChild(inputField); container.appendChild(submitButton); submitButton.addEventListener("click", () => { const value = inputField.value; resolve(value === "" ? null : value); }); } }); }35:[{"id":"053e846c2c085486b8a63ed257b4abb1a8ec85e69d8f117bc48df1d03c23c91eaa6c1124dbd2e3dbdcff8a770ce35432fbdce473fd7c518c38e553bdfc48ef37","href":"https://static.observableusercontent.com/files/053e846c2c085486b8a63ed257b4abb1a8ec85e69d8f117bc48df1d03c23c91eaa6c1124dbd2e3dbdcff8a770ce35432fbdce473fd7c518c38e553bdfc48ef37","name":"pie_schemas@1.json","type":"application/json","lastModified":1717765200656.868,"size":53150}] 2f:{"title":"Generations PIE - gold standard labeler","cells":[{"id":0,"value":"# Generations PIE - gold standard labeler","pinned":false,"hidden":false,"mode":"md"},{"id":243,"value":"## Step 1: Configure OpenAI API client\nEnter your OpenAI API key below and click submit.","pinned":false,"hidden":false,"mode":"md"},{"id":224,"value":"viewof openaiAPIKey = {\n let defaultValue = null;\n let keyDefined = false;\n try {\n defaultValue = localStorage.GPT_API_key;\n keyDefined = defaultValue != null;\n } catch (e) {}\n return Inputs.password({\n label: \"Enter your OpenAI key here:\",\n submit: true,\n value: defaultValue,\n width: 640,\n password: true,\n required: !keyDefined\n });\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":272,"value":"

","pinned":false,"hidden":false,"mode":"md"},{"id":245,"value":"## Step 2: Upload pathology reports and extracted labels\n\n1. Make sure to upload all pathology reports for which information was extracted by the two annotators.\n2. Upload the extracted JSON by the human (after standardization) and the AI. The order does not matter. \"Annotator A\" can either be the human-generated labels or the AI-generated labels. You also don't have to remember which one is which.\n3. Once you click Submit, the gold standard labeling process will commence.","pinned":false,"hidden":false,"mode":"md"},{"id":15,"value":"viewof gsData = guard(\n ({ template }) =>\n Inputs.form(\n {\n reports: Inputs.file({\n label: \"Pathology reports:\",\n accept: \".txt\",\n multiple: true\n }),\n labels1: Inputs.file({\n label: \"Annotator A labels:\",\n accept: \".json\",\n required: true\n }),\n labels2: Inputs.file({\n label: \"Annotator B labels:\",\n accept: \".json\",\n required: true\n })\n },\n { template }\n ),\n { submitLabel: \"Submit\" }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":151,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":177,"value":"\n\n

","pinned":false,"hidden":false,"mode":"html"},{"id":600,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":610,"value":"If you click the Refresh button below, it will display the gold standard table as it is being generated. This is what is stored inside your browser. Your browser won't forget it even after you shut down your computer. If you clear your browser cache, it will forget your progress.","pinned":false,"hidden":false,"mode":"md"},{"id":593,"value":"viewof refresh = Inputs.button(\"Refresh\", {\n label: \"Refresh view of gold standard:\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":528,"value":"goldStandardWIP = {\n // 'goldStandard' output cells is 2 cells below this. This is just a progress report.\n refresh;\n return localforage.getItem(\"goldStandard\");\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":596,"value":"goldStandardWIP","pinned":true,"hidden":false,"mode":"ojs"},{"id":583,"value":"## Step 4: Download gold standard 🥇\nAfter you are done with the gold standard labeling, your work panel should display a message of completion. After that the cell below will generate a tabular data. Download as JSON using the menu on the left of the cell below.","pinned":false,"hidden":false,"mode":"md"},{"id":535,"value":"goldStandard = generateGoldStandardLabels(\n await gsData.labels1.json(),\n await gsData.labels2.json()\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":85,"value":"# Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":251,"value":"openai = {\n const openai = await import(\"https://cdn.skypack.dev/openai@4.47.1?min\");\n return new openai.OpenAI({\n apiKey: openaiAPIKey,\n dangerouslyAllowBrowser: true\n });\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":83,"value":"localforage = import(\"https://cdn.skypack.dev/localforage@1.10.0?min\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":12,"value":"import { guard } from \"@mootari/inputs-submit\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":265,"value":"{\n const listModels = await openai.models.list();\n const apiKeyMessageContainer = document.getElementById(\"api-key-message\");\n if (listModels) {\n apiKeyMessageContainer.innerHTML = `

OpenAI API key is valid! 😊

`;\n } else {\n apiKeyMessageContainer.innerHTML = `

OpenAI API key is invalid. 🙁

`;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"getPathologyReports = async () => {\n const reportNames = await Promise.all(\n gsData.reports.map((file) =>\n file.name.split(\".\").slice(0, -1).join(\".\")\n )\n );\n const reports = await Promise.all(gsData.reports.map((file) => file.text()));\n\n return Object.fromEntries(\n reportNames.map((name, index) => [name, reports[index]])\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":74,"value":"function getType(value) {\n if (value === null) return \"null\";\n if (Array.isArray(value)) return \"array\";\n return typeof value;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":70,"value":"function valuesMatch(valueA, valueB, type) {\n if (valueA === null && valueB === null) return true;\n\n if (Array.isArray(valueB)) {\n if (valueB.length === 0) {\n return valueA === null;\n }\n return valueB.includes(valueA);\n }\n\n if (type === \"number\") return Math.abs(valueA - valueB) < 1e-2;\n return valueA === valueB;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":363,"value":"getTypeFromSchema = (schema) => {\n return schema.hasOwnProperty(\"type\")\n ? Array.isArray(schema.type)\n ? schema.type[0]\n : schema.type\n : null;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":210,"value":"displayHeading = (divID, varName, labelA, labelB) => {\n const container = document.getElementById(divID);\n container.innerHTML = `\n

Variable name: ${varName}

Annotator A label: ${labelA}

Annotator B label: ${labelB}

\n`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":253,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":117,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"$34","pinned":true,"hidden":false,"mode":"ojs"},{"id":106,"value":"# Danger zone!!!\nIf you want to restart building the gold-standard from the scratch, click \"Reset all progress\" button below. This is also useful if you made a mistake while uploading the pathology reports and the labels.","pinned":false,"hidden":false,"mode":"md"},{"id":89,"value":"viewof reset = Inputs.button(\"Reset all progress\", {\n label: \"Start from scratch?\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":98,"value":"reset ? resetProgress() : null;","pinned":false,"hidden":false,"mode":"ojs"},{"id":87,"value":"async function resetProgress() {\n reset;\n await localforage.removeItem(\"goldStandard\");\n await localforage.removeItem(\"lastKeyIndex\");\n await localforage.removeItem(\"lastRowIndex\");\n console.log(\"Progress reset.\");\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W35","stdlib":"1","titling":"auto"}