A Criticism of Visualization Criticism Criticism

Criticism in visualization can be harsh, pedantic, and stupid. But it is also a useful tool that shows the thinking behind the seemingly simple graphical shapes we use, and teaches people things they might not be aware of. While I largely agree with Andy Kirk's criticism of visualization criticism and the danger of scaring people away from visualization, his "grown-up criticism" argument cuts both ways: grown-ups can argue a point without getting upset.

It's not like all the web would be hostile to visualization experiments; far from it. Look at websites like infosthetics and flowingdata, which present links to lots of visualizations, good and bad, with little to no criticism. Now compare their traffic to what I am getting here, or what Bryan Connor's The Why Axis is getting, or Kaiser Fung's Junk Charts. Taken together, infosthetics and flowingdata have about ten times the RSS subscribers and 25 times the page views of this humble little blog (based on their sponsorship information here and here).

People like sharing pretty stuff to look at, and there are plenty of places to find things. Visualization is just another source of colorful pixels, just like photography, painting, collage, etc. The things that get shared are invariably beautiful, and sometimes they are also good visualizations.But more often than not, the question of whether something serves any kind of purpose, or is interesting in addition to being pretty, is not asked.

The Bike Analogy

Andy uses the example of a bicycle to argue that different people have different goals. A bicycle doesn't go very fast, it doesn't carry a lot of luggage, it doesn't work well in water, etc. It still has a lot of other good uses, though, and that's why people like bicycles. Fair enough. He's also right that criticism in visualization can be based on nonsensical assumptions or on reciting some oversimplified "rules."

But let's play with that bicycle metaphor a bit and consider a new bike design by a newcomer to the field. Our designer has looked at existing bikes and found that they all are too similar and not nearly as exciting as they could be. So here is a new design! It's pretty, it's intricate, and it uses ideas from other fields. I hope you like it.

This is what a lot of visualization experiments look like, and that's fine. But it has to be possible to criticize them. This is not a useful bicycle. It has other virtues, perhaps, but if my goal is to get from point A to point B, this is not going to do it. The fact that you can actually ride it and it works makes not difference here, it's not a practical bike. As a decorative piece, it's clearly superior to a lot of more useful bikes, though. But is bicycle design, as a field, about decorative pieces with no practical value, or perhaps more about efficient modes of transportation? What should be the typical assumptions and goals of bicycles, and which goals might be somewhat less important?

I Scare Because I Care

Visualization is still very young, and there are many influences and potential directions. That's great, and we should embrace that. But at the same time, we also need to figure out where the boundaries are. Without boundaries, there is just chaos. People often do things and call them visualizations even when they're not. They don't know how to use colors. They don't know what perception is. They've never heard of scaling or baselines or part-whole relationships.

Different people also have different ideas about what they want from visualization, but don't realize it. There is much confusion about what visualization is, what its goals are, etc. But that is exactly why we need to clarify, rather than muddle up and sweep under the rug. We need to point to things and figure out why they are different. And sometimes, somebody needs to play the curmudgeon and lay down some rules.

Because this field is so young, it needs a culture of discourse. We need to be able to argue. And we need to be able to criticize. Without that, there is no progress and no evolution, only chaos.

Posted by Robert Kosara on July 9, 2012. Filed under criticism.

---

Comments

Dino Citraro says…

I believe William Playfair might disagree with your assessment of visualization being very young. 😃

Data visualization methods such as the bar chart, the line graph, the box plot, etc, are unique because when used properly, they visually describe the data in a simple and elegant way. They are efficient. When people break the conventions of how traditional methods are used, it is easy to want to criticize, since something elegant and efficient has been made less so. However, if the message is being conveyed, regardless of how, then we should applaud the new method and learn from it.

We shouldn't feel the need, however, to blankly call it a data visualization, or assume it is even closely related.

Hector Cabrerra says…

chaos itself can be evolution of the visualization. i dont think we should see it as a bad thing. some people do it right, and some dont, and even dosnt care about it. its the way it is, ans its the only way that we can proceed.

Carlos Scheidegger says…

Amen! If anything, the culture is too nice! 😃 I see criticism mostly being leveled at practitioners, and not at the academic production in the field. Have you seen the debates on stats blogs about bayesianism or CS theory (or, gasp, theoretical physics?)

Readers of every criticism post should insert the following implicit prelude: "look, I love you enough to post about this; if I hated this work as much as you might fear I do, I wouldn't even give it the time of day." and an epilogue to the tone of "Let's have a group hug, sing a song by the fireplace. Ok, done? Now we all know we love you. But we need to talk about the work."

Why do people feel so personally threatened by discussion? It's crazy. Part of me suspects it's insecurity stemming from some misconceived notion that there's no intellectual depth in the field.

Ali says…

Could we model the spectrum of projects that visually display data as one that extends along two axes: effectiveness (ability to provide insight and afford decision-making) and, say, aesthetic appeal (impact of the visual artifacts)? That would then give us a category of visualization that we could term 'art' whose sole purpose would be to impress without necessarily being insightful or informative.

I can think of several visualizations, usually network-based ones, that would fall into that category, as you point out.

Santiago says…

I don't think Andy tries to reduce criticism relevance. He just says that it needs to be better (more clear, balanced…)

Hamilton says…

"It has other virtues, perhaps, but if my goal is to get from point A to point B, this is not going to do it."

It looks usable to me. Why again won't this do it?

Al says…

Re. the comments about Bayesianism, CS theory and theoretical physics: An important difference between us and them is, they are mature fields which demand as high standards of their criticism as they do of their primary work. Sloppy criticism is criticised as harshly as sloppy research.

Much of what passes as criticism in our field is little more than knee-jerk reactions. It's not criticism if it doesn't consider a piece based on what that piece is actually trying to achieve.

We should have higher standards, and should demand better from our criticism. Preaching one author's opinions and tastes from the '80s as if it is gospel, quoting made-up rules or jargon-laden yet substance-free generalisations or, worst of all, "correcting" someone elses' work without even pausing for 5 minutes to think about what that work's actual real brief was, are not examples of useful criticism - but noise like this is common and goes unchallenged.

At bare minimum, a critique in ANY field should demonstrate appreciation of the actual problem a piece is trying to solve, as well as (as Andy Kirk says) a "smarter appreciation of the different contexts in which these works are created". For published visualisations, this will almost always involve considering the target audience (their prior knowledge, level of interest, their motivations and interests...), and the piece's intended distribution.

Yet I almost never see this in visualisation criticism - so often, the critics arrogantly assume that the intended target audience is themselves, and that the intended aim is giving them the data to analyse their way. This is a crazy assumption. Why would any published visualisation be aimed at the one audience (analysts) who don't need the published visualisation and who could analyse, visualise and make sense of the data just fine without it?

We need to demand more genuine, valid, useful criticism, and tolerate less noise.

Al says…

I really don't understand the use of the bike metaphor in this article. He seems to be saying that there should be a universal set of a-priori assumptions about what visualisations are for, which should trump case-specific context - like how there is for bikes. But there isn't, even for this straw-man example.

Take the argument to its logical conclusion, and we'd have Mr Kosara marching in to a Saatchi gallery to lambast the owners of this bike for its being ill suited for commuting to work (or some other supposed universal purpose of all bikes). Apparently, the gallery owners should keep a straight face, sagely nod and take this as serious, useful criticism, because this generalised assumption trumps even the massively obvious clues as to the very different intended purpose of this specific bike/exhibit.

Criticise each experiment on the terms and scope of that specific experiment. Here, it's "Do people react if I make a bike out of a metal fence?", and the answer is "Enough for it to be exhibited by Saatchi".

p.s. re "What should be the typical assumptions and goals of bicycles, and which goals might be somewhat less important?" - Who is the bike for? Commuters, downhill sports, racing, stunts, kids, shopping, long distance riding, off road travel? Pedal powered, fuel motor, electric motor, a combination? Just one rider, two riders, one rider and passenger?

Structural integrity, balance, effective breaks and steering, and controllable motion that is faster than walking are probably universal goals between all of these established purposes of bike design, but there is little more than that. For each of the above purposes, there exists a successful, popular design of bike that fails for all the other above purposes. Anyone using this as criticism (for example, saying a stunt BMX fails by lacking any basket for carrying groceries, or that a fold-up commuter bike fails when taken down hill at speed) would simply look silly.

clay says…

The notion of constructive critique is pretty well understood by several non-science knowledge domains, e.g., design, specifically architecture, product design, etc.

A common approach is to analyze a design using a semiotics of form, specifically:

Semantics: what the form means (think, a button “means” object to be pushed, a doorknob “means” an object to open doors with – a round fuzzy thing means soft, a sharp pointy thing means ouch)

Syntactics: how the form interacts with similar other forms (think, if I were to sit all the toasters in the world in a line, what comparisons can I draw? Does this one toaster “fit” with all the other toasters? Is it different in a way that informs the semantics? Or is it different for the sake of being different? This often requires a distinction between vagueness and abiguity, where vagueness is absence of meaning and ambiguity is conflicting meaning.)

Pragmatics: what the form affords (and this is a specific definition of affordance, as in, stair cases afford humans the ability to move between floors in a building, under their own power – but notice I didn’t say utility… it’s an important distinction, but gets even further into the weeds…)

What the data viz community is missing, in my opinion, is that it’s not a style versus utility debate, but that the “style” is determined by the combination of meanings. And this is not a new idea:

http://en.wikipedia.org/wiki/Visual_semiotics (aside: that’s a really crummy Wikipedia page but you can get the general idea…)

This is not to be boiled down to “form follows function” – that’s way too simple (and overused… and distinctly modernist and boring).

If you start to consider “rating” an infographic using semiotic analysis, you can really understand and specifically critique things that you can't get to by asking whether an infographic is “stylish” or “utilitarian”. Give it a try – take any data viz or infographic and ask it: are you helping me make sense of something and take action? How do you fit with your peers? What are you supposed to mean?

In many ways, I believe we're wired to shortcut this type of analysis to speed up our decision making process - and thus, we often revert to "taste" and "style" when critiquing matters of meaning and sense making.