Memorability, Science, and The Value of Thinking Outside the Box
Stephen Few has written a long and scathing piece tearing down a paper presented at VIS earlier this year. While some of his criticism is justified, it is too focused on one of the authors, and it comes from an idea of visualization research that is far too limited.
The paper in question is Beyond Memorability: Visualization Recognition and Recall by Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva.
There is a discussion thread that a number of visualization researchers have posted in to respond, including Jean-Daniel Fekete, Jeff Heer, and Ben Shneiderman. I therefore won't attempt a detailed response here. Instead, I hope to illuminate the way visualization research works, and whether or not it is a science.
Steve’s main concern is what he considers the lack of scientific rigor in the paper and the field in general. He complains about several issues with the paper, including the small number of study participants, the study mechanism of showing people images for a short time, etc. This is the part I won’t respond to in detail because it just gets too technical and has already been largely addressed by others. Instead, I will try to talk about research in visualization in a more general way.
Why the Personal Attacks?
But let’s get this out of the way first. The whole thing is centered on Michelle Borkin, even though the paper has eight authors. At one point, Steve hypothesizes why the quality of the paper is so bad:
I suspect that her studies of memorability were dysfunctional because she lacked the experience and training required to do this type of research.
I can see where he’s coming from. Michelle got her Ph.D. at a little place called Harvard. Do they teach the scientific method at Harvard? I don’t know. It’s only ranked second in the U.S. overall and third globally for both Computer Science and Physics. Though perhaps we could ask the dozens of Nobel Laureates that work there or came from there.
The statement above is especially silly because Michelle’s Ph.D. is actually in Applied Physics, which in contrast to Computer Science is an actual science. It is also odd because Steve praised one of her papers in 2011. If that was a scientifically sound paper, has she forgotten the scientific method in the meantime? Has the exposure to all those infographics gotten to her?
Another problem is that Steve has so far mostly singled out women for his criticisms: Jessica Hullman in 2011, Michelle Borkin in 2013, and now again in 2015. Jeff Heer also picks up on this, and in particular the personal attacks, in the discussion thread:
Posted by Robert Kosara on December 16, 2015.