Appearance

Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them ​

Replications are rare in visualization research, but if they were more common, it is not unreasonable to believe that they would show a similar rate of unreproducible results as in the psychological and social sciences. While a replication crisis in visualization research would be a helpful wake-up call, examining and correcting the underlying problems in many studies is ultimately more productive. In this paper, we survey the state of replication in visualization research. We examine six threats to the validity of studies in visu- alization and suggest ways to address them. Finally, we describe possible models for publishing replications that satisfy the novelty criterion that can keep replications from being accepted.

Robert Kosara, Steve Haroz, Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them, Proceedings BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV), 2018. DOI: 10.1109/BELIV.2018.8634392
bibtex
@inproceedings{Kosara:BELIV:2018,
	year = 2018,
	title = {Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them}, 
	author = {Robert Kosara and Steve Haroz}, 
	booktitle = {Proceedings BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)}, 
	doi = {10.1109/BELIV.2018.8634392}, 
	talk = {https://vimeo.com/305865159}, 
	abstract = {Replications are rare in visualization research, but if they were more common, it is not unreasonable to believe that they would show a similar rate of unreproducible results as in the psychological and social sciences. While a replication crisis in visualization research would be a helpful wake-up call, examining and correcting the underlying problems in many studies is ultimately more productive.  In this paper, we survey the state of replication in visualization research. We examine six threats to the validity of studies in visu- alization and suggest ways to address them. Finally, we describe possible models for publishing replications that satisfy the novelty criterion that can keep replications from being accepted.}, 
}