Watchlist: Jessica Hullman
Among the papers that stood out at InfoVis 2011 were two that shared an author, and that were presented in the same session by the same person: Jessica Hullman. These papers were Benefitting InfoVis with Visual Difficulties (with Eytan Adar and Priti Shah) and Visualization Rhetoric: Framing Effects in Narrative Visualization (with Nicholas Diakopoulos).
Now I admit that I have a particular interest in both topics: cognitive issues in visualization and visual rhetoric/storytelling. But both topics are increasingly recognized as important by others in the field, which explains not only why the papers were accepted, but also the Best Paper Honorable Mention for the Visual Difficulties paper.
The Visualization Rhetoric paper describes the use of visualization in the telling of news stories. While we like to think of data visualization as an objective science, there are many subtle and not-so-subtle ways in which a visualization can be framed and manipulated to tell different kinds of stories. The paper describes some of these choices in examples and touches on many issues that are not typically considered in visualization, like connotations, cultural codes and conventions, etc. It's a great addition to Segel and Heer's storytelling paper from 2010 and a treasure trove of ideas and issues to think about when looking at how visualization is used in the media.
A recent paper on learning found that making tasks more difficult can be beneficial for learning. In particular, the use of difficult fonts can improve people's retention of facts from a piece of text. The Visual Difficulties paper examines how this idea might apply to visualization. This is clearly a speculative paper, and the authors are careful to discuss issues with findings like preferences for certain visualizations versus people's actual performance using them. The entire idea is counter-intuitive to begin with, but the ideas presented and the entire direction of the paper are intriguing. What the paper lacks are concrete designs or experiments that show how difficulties actually benefit people using a visualization, though I have no doubt that somebody will conduct such a study soon.
In addition to the papers themselves, the presentations were also remarkable: Jessica referred the audience to the papers for more details. While it sounds trivial, a common mistake is trying to pack every little detail of the paper into a 15-minute presentation, rather than giving a good motivation and clear, concise idea of the work.
And Now A Long Defense Against Stephen Few
In his October/November/December 2011 Visual Business Intelligence Newsletter, Stephen Few attacked Hullman's Visual Difficulties paper at some length. Unfortunately, many of his arguments seem to be based on a superficial reading of the paper that glosses over some important details of the arguments the authors are making.
He essentially opens his critique with a large picture of a 3D pie chart, criticizing Hullman et al.'s endorsement of 3D charts. What they say, however, is a lot more nuanced. They clearly report findings that show that 3D charts typically decrease performance (Section 1.2.1). The point they make is about preference and engagement, and is clearly phrased as a possibility and not a recommendation.
Steve also dissects their supposed misunderstanding of chart junk. This is largely based on what is clearly a typo in the beginning of Section 3.1.2 in the paper: "Charts with higher data-ink ratios have conventionally been equated with ‘embellishment’ or ‘decoration’." It's obvious from the rest of the paper that the authors understand what chart junk is, though, and they correctly use the data-ink-ratio in other parts of the paper, including a few sentences later in the same paragraph as the incorrect use.
Another point of criticism is the question of metrics. When Hullman et al. use a study on people's preferences as an argument for 3D charts. They do not claim that they are more effective, as Steve suggests, but correctly talk about preference. In fact, they clearly contrast preference to performance, stating that performance is in fact decreased by 3D effects.
Apart from Steve's somewhat superficial reading of the issue, the underlying question is a really interesting one: how should we measure effectiveness? The easy answer is to look at accuracy and response time, which has been the gold standard so far. But is that really all there is? Hullman et al. also report on a study by David Sprague and colleagues that looked at a music visualization and describe the contrast between people's preferences and performance. Participants in Sprague's study performed worse using an animated version of the visualization, but still clearly preferred that. Stephen Few quips that, "[as] we all know, people too often prefer things that aren’t good for them." – but I don't think it's quite that simple.
If we assume that analysis is the only way people use visualization, then perhaps accuracy and task completion time are the only metrics we should care about. But I see a lot of people do things other than analysis, and I'm starting to think that analysis is really just a small part of the bigger picture. When people get together to look at a visualization to discuss some data, they do many things in addition to (and on top of) analysis: storytelling, arguing, questioning, building scenarios, pointing to other visualizations, drawing hypothetical charts with their fingers, etc. There is no question that analysis is important, but it's not nearly all there is.
The power of visualization is not that it's such an accurate representation of data; in fact, there are many ways to represent data more precisely than using visualization, like a simple table of numbers. What visualization brings to the table is a much more holistic view of data, which a table simply cannot provide. The uses for this modality are consequently a bit more complex and varied. I also question Steve's assertion that recall of visualizations is not needed. I see a lot of workbooks here at Tableau that people look at every day or every week as part of their work, and they definitely want to be able to remember what they saw last time. Otherwise, what would be the point of creating them? They won't remember details, but they certainly know whatever features are important to them, stood out recently, etc.
I'm not saying that the paper is flawless, because it isn't. But it is an intriguing step into an entirely new direction. There is more work that needs to be done, more evidence that needs to be collected, and more papers need to be written. But the paper is not nearly as flawed as Steve makes it appear. I, for one, am looking forward to seeing more of Jessica's work.
Posted by Robert Kosara on February 8, 2012. Filed under journalism.