Memorability, Science, and The Value of Thinking Outside the Box

Stephen Few has written a long and scathing piece tearing down a paper presented at VIS earlier this year. While some of his criticism is justified, it is too focused on one of the authors, and it comes from an idea of visualization research that is far too limited.

The paper in question is Beyond Memorability: Visualization Recognition and Recall by Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva.

There is a discussion thread that a number of visualization researchers have posted in to respond, including Jean-Daniel Fekete, Jeff Heer, and Ben Shneiderman. I therefore won't attempt a detailed response here. Instead, I hope to illuminate the way visualization research works, and whether or not it is a science.

Steve’s main concern is what he considers the lack of scientific rigor in the paper and the field in general. He complains about several issues with the paper, including the small number of study participants, the study mechanism of showing people images for a short time, etc. This is the part I won’t respond to in detail because it just gets too technical and has already been largely addressed by others. Instead, I will try to talk about research in visualization in a more general way.

Why the Personal Attacks?

But let’s get this out of the way first. The whole thing is centered on Michelle Borkin, even though the paper has eight authors. At one point, Steve hypothesizes why the quality of the paper is so bad:

I suspect that her studies of memorability were dysfunctional because she lacked the experience and training required to do this type of research.

I can see where he’s coming from. Michelle got her Ph.D. at a little place called Harvard. Do they teach the scientific method at Harvard? I don’t know. It’s only ranked second in the U.S. overall and third globally for both Computer Science and Physics. Though perhaps we could ask the dozens of Nobel Laureates that work there or came from there.

The statement above is especially silly because Michelle’s Ph.D. is actually in Applied Physics, which in contrast to Computer Science is an actual science. It is also odd because Steve praised one of her papers in 2011. If that was a scientifically sound paper, has she forgotten the scientific method in the meantime? Has the exposure to all those infographics gotten to her?

Another problem is that Steve has so far mostly singled out women for his criticisms: Jessica Hullman in 2011, Michelle Borkin in 2013, and now again in 2015. Jeff Heer also picks up on this, and in particular the personal attacks, in the discussion thread:

Posted by Robert Kosara on December 16, 2015.

---

Comments

Monika M. Wahi, MPH, CPH says…

Thank you for calling out that vitriol; it was uncalled for. That said, Harvard indeed does not generally teach the scientific method, in my experience. When I tutor people from Harvard or work with graduates, I have to teach it to them. It is a serious problem and has confounded the progression of public health science.

I also call into question the visualizations presented in the article of their results. When one visualizes non-significant results as was done in Borkin et al., it confers the implication they are important for some reason. I had trouble seeing the "importance" of what was visualized - without a clearly stated hypothesis (or objective), it is not clear what the team found.

In terms of a visualization, I remember Marshall McLuhan's quote, "The medium is the message." Simply making a visualization suggests that what is in the visualization is important, and the visualizations presented in this article (their bar graphs, and their HR graphs) suggest that these results are meaningful enough to deserve to be visualized. I respectfully disagree given the poor description of the study and questionable study design.

This is unfortuantely typical in science, and singling out this particular article surprises me. I would say about 80% of the articles I read go something like this. Results of unclearly described, poorly-designed studies are overinterpreted by unnecessarily large lists of scientific authors and somehow published in the peer-reviewed literature. Harvard authors get a free pass from other reviewers, but I don't give them one. With all, I insist on clarity and clear communication that speaks to multiple disciplines about what the point was of the article. I'm a Charles Hennekens fan. What did you seek to find? Did you find it? This article does not answer either of these questions with their text or visualizations, and their results are therefore not actionable.

Jordan G says…

I like Stephen Few. A very much like how he's not afraid to challenge what's commonly accepted. At the same time, I too felt his most recent newsletter read like a diatribe against one researcher more than it did a critique of current research in data visualization. I'll grant the paper my Borken et al could have done certain things differently or better, I suppose. I'm simply acknowledging that concern because others have said it though I have no specific suggestions. To me, that they could have done this or that better is not all that important, and in reality it can be said of all research.

What's more important is that the researchers in this case sent their work in for peer review through a time-tested albeit admittedly imperfect system of scientific review. And though we can argue about whether this research should have passed review, we can all hopefully agree the researchers in this case participated in the traditional scientific process of having their work vetted by experts and peers. I bring this up because Few has NOT done any of this. His critique of the research was not sent anywhere for a double-blind review. He did not receive expert advice on what might make his work better. He did not have to address challenges from reviewers. In other words: he did not submit his own work for scientific scrutiny.

That's important. Because to the extent he is challenging others to more rigorously follow the scientific method, he should open himself up to the same type of scrutiny to which the researchers opened themselves. I'm not saying he doesn't open himself up to scrutiny personally. He regularly invites lively debate of his work. But his critique is not on the same playing field as a critique that would have gone through a double-blind review process and was subsequently published by the same journal.

Bottom line: I don't think it's fair to accuse your opponent of a lack of traditional scientific rigor when your response circumvents the traditional process of scientific scrutiny. Calling research out in a newsletter is not an example of scientific critique.

Jessica Hullman says…

Lots of points I agree with here (that is probably not a surprise 😃. But I'm not sure its fair to say that Few singles out women if you look at his critiques overall. He's done the same to multiple male-first-authored works.

Eytan Adar says…

Monika, if I'm understanding you correctly, I think that's a misread of McLuhan. In fact, what McLuhan suggests is that the artifact (i.e., the visualization medium) is worth studying independent of the content which it conveys, as it has influence (the message of the medium as distinct from the content communicated over the medium)*.

Visualizations of this type exist, and will continue to exist. Saying that they should not, will not make them vanish. Given that they do exist, isn't it appropriate to study their effects, why people make them, and why their designers have adopted certain strategies? (good and bad). Few's position, as far as I can tell, is that all such visualization are (capital-b) Bad (for his business? for the world? I don't know), all research that implies otherwise must be wrong, and any academic discipline that would allow the existence of such work to be corrupt. Once committed to that initial "this type of infovis is Bad" position, it becomes impossible to understand the study for what it is.

• I should admit that as many times as I've tried, I've never been able to make it the whole way through "Understanding Media", but I think this is a good summary (from: http://individual.utoronto.ca/markfederman/article_mediumisthemessage.htm):

Thus we have the meaning of "the medium is the message:" We can know the nature and characteristics of anything we conceive or create (medium) by virtue of the changes - often unnoticed and non-obvious changes - that they effect (message.) McLuhan warns us that we are often distracted by the content of a medium (which, in almost all cases, is another distinct medium in itself.) He writes, "it is only too typical that the "content" of any medium blinds us to the character of the medium." (McLuhan 9) And it is the character of the medium that is its potency or effect - its message.

Jamie Briggs says…

Being a long time fan and follower of both you and Stephen, I have to say, I see an awful lot of strawman arguments here, and a lot of emotional, in-substantive response.

I also see a completely ludicrous and uncalled for claim that Stephen singles out women in his critiques - you start by saying that Stephen's tone crossed the line. Your completely untrue accusation that Stephen is attacking women crossed the line, Robert. I think you're better than that.

I also see a great deal, in this article and many other responses, of what can only be called tone policing.

Yes, Stephen's critiques and even his every day conversation can come across as harsh. His style is frank, direct, and sometimes lacking in sensitivity.

But I don't see anything in his critique or in his responses to it that is emotional, personal, or vitriolic.

Stephen writes in a style that will alienate some people, and it would benefit him to recognize this. But let's focus on the actual critique and its content, rather than on his perceived tone.