Paper: Skipping the Replication Crisis in Visualization
Visualization doesn't have the replication issues that some other fields are struggling with right now, but is that because our science is so strong or because nobody actually bothers with replications? And what can we do to get ahead of potential problems before we run into a full-on crisis? In a paper to be presented at BELIV, Steve Haroz and I list potential pitfalls and present possible solutions.
Replication is a part of the scientific process, and in some of the more established sciences like physics, it's a given that any single finding needs to be repeated by others before it is accepted. Over the last few years, researchers in fields like psychology have found that they were unable to produce similar results when repeating experiments, leading to what is called the replication crisis: if many studies only produced an effect once, how much of what we think is true is just due to chance or mistakes (to say nothing of data manipulation or fraud)?
Visualization isn't all that different from psychology. There are very few replications (and the few that are done are very difficult to get published), and the way we like to work with data easily leads to cherry-picking and other problematic practices. Are we on the verge of a replication crisis – or would we be if anybody bothered to replicate experiments in visualization?
Steve Haroz and I look at six different sources of problems, from bad study design to misinterpreted results, describe why and how they happen, and what can be done about them. We also discuss a number of ways replications can work, from direct replication (same experiment) to conceptual replications (same phenomenon, but different experiment) and registered reports (which get reviewed before the experiment is run to minimize p-hacking).
One reason why replications are hard to publish in our literature is that they are not considered novel. We therefore also propose a few ways of working replications into papers to make them more palatable for reviewers, though we also argue for a more scientific publication landscape in visualization.
The paper will be presented at BELIV, which is part of IEEE VIS, on Sunday, October 21. We're in the first paper session after the keynote, and Steve also has a related paper in the session after that one. Even if you're not coming to VIS, you can click the link below to read the paper.
Robert Kosara, Steve Haroz, Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them, BELIV 2018.
Posted by Robert Kosara on October 16, 2018. Filed under paper, papers.