Do Mechanical Turks Dream of Square Pie Charts?

Online studies are an attractive alternative to the labor- intensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon’s Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one of which was identical to a previous study performed in our lab. We report on our experiences and present ways to avoid common problems by taking them into account in the study design, and taking advantage of Mechanical Turk’s features.

Robert Kosara, Caroline Ziemkiewicz, Do Mechanical Turks Dream of Square Pie Charts?, Proceedings BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV), pp. 373–382, 2010.

@inproceedings{Kosara:BELIV:2010,
	year = 2010,
	title = {Do Mechanical Turks Dream of Square Pie Charts?}, 
	author = {Robert Kosara and Caroline Ziemkiewicz}, 
	booktitle = {Proceedings BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)}, 
	pages = {373–382}, 
	abstract = {Online studies are an attractive alternative to the labor- intensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon’s Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one of which was identical to a previous study performed in our lab. We report on our experiences and present ways to avoid common problems by taking them into account in the study design, and taking advantage of Mechanical Turk’s features.}, 
}