31:T2086,

Replication Study - V1

See also Replication Study - V0 for history.

Outcomes

Undertake an exact, close or conceptual replication of an existing study;
Interpret the methods used in the original study and critique with a view to close and conceptual replication;
Develop the knowledge, skills and resources required to replicate;
Operationalise a study that offers an exact, close or conceptual replication;
Compare the results of replication studies with originals to develop knowledge;

Consider a focus on values, attitudes and skills outcomes. Can include knowledge outcomes where necessary, but useful to think about emphasising process as part of the constructivist approach

What's the point?

Provide a scaffold for students to undertake some original research as part of their training;
Gain experience of the ways in which research is conducted;
Assess the role of replication as a means of assessing the reliability and value of research claims;
Assess the importance of open scientific reporting as a mechanism for achieving quality (and inviting scrutiny);

${bib.cite("schwab_how_2023")} describe replications as providing rich experiences for doctoral students, in which general general research skills are developed as students "strive" to imitate "good" existing studies.

Useful to ask yourself why is this module necessary / valuable for the CDT training

Where's the knowledge constructed?

Cognitive?

Students will collate and generate their own materials for the replication;
Students will describe the study and make decisions on and applications for ethical approval;
Students will undertake analysis and make comparisons with existing stuides, generating artefacts that capture their processes, analysis and reflection

Social?

Students will work in groups on the replication Study
Groups may work in pairs - to compare the replications

We are adopting a constructivist approach to teaching. This includes cognitive constructivism and social constructivism. Both encourage learner building of knowledge but require scaffolding to be successful.

Artefacts

A planned and implemented experiment that replicates existing research;
A critical comparison between original results;
Reflection on the replication experience and comparison, with thoughts and learnings on scientific reporting and generalization;

What will the students produce? What will they be able to comment upon in their PhD Pathway document?

Plan

Read the papers
Create groups - select papers
Decide on form of replication
Ethical approval - request & process
Develop instrument
Run pilot - with pilot analysis
Recruit participants - Prolific
Run study - collect data
Data analysis
Reflection, discussion & writing

What kind of activities outside of the timetabled sessions are you expecting students to do?

Feedback

Feedback opportunity 1
Feedback opportunity 2
Feedback opportunity 3 etc.

Where and how does feedback occur? (Because that’s the important bit of “assessment”) This can (should) be formative and summative (post artefact, post module).

Module topics

Vis Repligogy : Towards a Culture of Facilitating Replication Studies in Visualization Pedagogy and Research (Syeda et al, 2024)
How replication studies can improve doctoral student education ${bib.citep("schwab_how_2023")}
Exact, Close and Conceptual Replication (Schmidt, 2009)
- exact replications are conducted by the original authors [Sch16];
- close replications are independent replications by other researchers aiming to replicate the original study as closely as possible [BID 14];
- conceptual replications build upon the original research [Sch16];
Selecting a Study
Ethics and Approval
Designing a Replication

See Zotero List for papers - add comments

What will the module cover? May be cross-cutting themes or session-focussed topics.

32:T569,## Approach - JD Suggestions Students need to do most of the production here, we are guides ('check-in'). #### Questions - How to engage with **VIS Repligogy** (_Michelle Borkin_ approached at VIS)?
**CT**: Kick off / inspiring talk from Michelle (30/30)? - How to engage with [**ReVISiT**](https://revisit.dev/) (_Alex Lex_, _Lane Harrison_ approached at VIS, discussed 02/04 and will host workshop 29/04)?
**CT**: Tech / provenance talk from AL / LH? Chidi MasterClass? #### Answers - How to use **VisUnit** (RJ)? - 33:[] 2f:{"title":"Replication Study ","cells":[{"id":0,"value":"#

Replication Study

","pinned":false,"hidden":false,"mode":"md"},{"id":10,"value":"$31","pinned":false,"hidden":false,"mode":"html"},{"id":367,"value":"## Plan\n\nSee [Replication Study Plan 2026](https://observablehq.com/@jsndyks/replication-study-plan-2026)\n","pinned":false,"hidden":false,"mode":"md"},{"id":214,"value":"## Papers to Replicate ...","pinned":false,"hidden":false,"mode":"md"},{"id":315,"value":"See [Protocol Close Reading](https://observablehq.com/d/a4d50143ea945cd3) list ...\n\n#### Cleveland & McGill \n\nThe students have been introduced to ${bib.cite(\"cleveland_graphical_1984\")} a few times.\n\nJo's wonderful [Stimulus Simulator](https://observablehq.com/@jwolondon/stimulus-simulator) was produced when we discussed this over Christmas - it's superb and give us plenty of traction on replication and options for studying it.","pinned":false,"hidden":false,"mode":"md"},{"id":233,"value":"false\n ? md`---\n\n#### BallotMaps \n\nBallotMaps ${bib.citep(\n \"wood_ballotmaps_2011\"\n )} would be great - things to consider:\n - Does _HiDE_ work or can we use _Tableau_ or _plot_?\n - redesign of graphics?\n - more parties (_Reform_?)\n - do we need [_OnoMAP_](https://onomap.org/) - still exists!\n - can we add a more thorough (systematic) search for related material (and re-read of related material) ... including stuff that has been written since?\n\n[London Elections in 2026](https://en.wikipedia.org/wiki/2026_London_local_elections) are on 7th May.`\n : md`--- \n_BallotMaps was Here_`","pinned":false,"hidden":false,"mode":"ojs"},{"id":259,"value":"false\n ? md`---\n\n#### Sketchy Rendering \n\nSketchy ${bib.citep(\n \"wood_sketchy_2012\"\n )} is something we could do in _plot_ - things to consider:\n - **low level** - redo stats from a frequentist perspective?\n - change parameters and examples?\n - use a panel (Prolific)\n - **high level** - straightforward, different stimuli?\n\nNote that people are more exposed to this kind of thing, and perhaps more skeptical now?\n\nSee [ReVISit - Sketchy Rendering](https://observablehq.com/@jsndyks/revisit-sketchy-rendering)`\n : md`--- \n_Sketchy was Here_`","pinned":false,"hidden":false,"mode":"ojs"},{"id":279,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":290,"value":"----\n\n### JD's Notes ...\n\nPriorities:\n - find studies (and get students to read)? _Sketchy_, _BallotMaps_\n - introduce different perspective?\n - read replicology?\n - read replication type paper?\n - contact ReVISit (Lex), VisUnit (Radu), Borkin?\n - set up Prolific?\n","pinned":false,"hidden":false,"mode":"md"},{"id":296,"value":"notes = task(\n md`\nHelp!\n\nWe need a hand with a replication study on the stats side of things.\n\nThe replications we envisage involve using NHST and I while I can oversee, I don’t have the expertise to lead this reliably.\n\nDoes anyone have the capacity and capability to oversee some NHST analysis this summer?\nWe’d need you for a couple of classes and on hand to supervise analysis on occasion.\n\nThe tests used in the original studies are …\n\npaired and unpaired T-tests\n(with Bonferroni correction - which is easy to apply but I am mush less confident about justifying it!)\nrepeated measures ANOVA\nMann-Whitney U tests\n\nJason.`,\n { title: \"EMail Draft - Statistics Expertise Required\" }\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":350,"value":"#### Also ...\n\n*Replication Study Notes:*\nhttps://observablehq.com/@jsndyks/replication-study-a3feb525a78e93c9\n\n*Experiment Papers:* \nhttps://observablehq.com/@cdt2025/data-vis-research-08","pinned":false,"hidden":false,"mode":"md"},{"id":5,"value":"

\n\n---\n\n## Appendices","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"import { cardStyle, setFonts, setPageStyle } from \"@jwolondon/cdt-utilities\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":172,"value":"import { Refs } from \"@gicentre/references-class\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":135,"value":"setFonts()","pinned":false,"hidden":false,"mode":"ojs"},{"id":126,"value":"cardStyle()","pinned":false,"hidden":false,"mode":"ojs"},{"id":226,"value":"setFonts()","pinned":true,"hidden":false,"mode":"ojs"},{"id":228,"value":"setPageStyle()","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"","pinned":false,"hidden":false,"mode":"html"},{"id":165,"value":"bib = Refs.create(\n await d3.text(\n \"https://jsndyks.github.io/diverse-cdt/bib/zoteroDIVERSE-CDT.bib\"\n )\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"_Reference styling ..._\n\n```css\n.csl-entry {\n line-height:1.3 !important; padding-bottom:0.5em; font-size: 90%;\n}\n.csl-entry .csl-author {\n font-weight: bold !important;\n}\n```","pinned":false,"hidden":false,"mode":"md"},{"id":273,"value":"import {\n aims,\n info,\n annotation,\n task,\n learning,\n box,\n designNote\n} from \"@jwolondon/schema-teaching2025\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}