Tufte and the Truth about the Challenger
Almost exactly 25 years ago, on January 28, 1986, Space Shuttle Challenger disintegrated seconds after lift-off. One of Edward Tufte's most famous examples of bad charts are the ones used by engineers who argued against the launch, and who failed to convince. It's a fascinating story, but it has one major fault: it is not true.
The Charge
While it is easy to look at this as purely a question of proper data display, the claim Tufte makes is a very serious one: the engineers at Morton Thiokol, which manufactured the rocket boosters that led to the disaster, failed to communicate their concerns, and were thus partially responsible for the death of seven astronauts.
The authors of the paper this posting is based on, Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger (details below), claim that Tufte both misunderstood and misrepresented the argument and the true cause of the failure. What is more, Tufte keeps making the case that it was the engineers's fault, despite the fact that information is readily available that shows what really happened.
O-Ring Failure
To briefly summarize the issue, what happened was that the aft joint on one of the Challenger's rocket boosters failed during launch, which led to the escape of hot gases to the side, towards the main fuel tank. That tank was quickly ruptured, and started to disintegrate. The tank and shuttle were then torn apart by aerodynamic forces (the Wikipedia article argues that there was no actual explosion).
Tufte has argued that the main cause for the joint failure was damage to the O-rings, which sealed the joint to prevent gases from escaping through a gap between parts of the rocket. That failure was supposedly due to the fact that cold temperatures (colder than any shuttle launch before) had led to a loss of flexibility in the rubber-like material the O-rings were made from.
The main argument in the paper is that the cause of the failure in the joint was not in fact O-ring damage, but blow-by: hot gases passing by the O-rings when they do not properly seal. Also, Tufte's elegant chart showing temperature versus damage is misleading, because it does not show the actual O-ring temperature, but a mix of O-ring and environmental temperature (which can differ significantly).
Physics
Tufte describes Richard Feynman's famous demonstration of the O-ring's lack of flexibility after having been compressed in a glass of ice water. He criticizes the lack of a control on this experiment, and that Feynman did not make clear what he was comparing (temperature, wet conditions, etc.), and to what reference. But a very similar experiment, with controls and with a much better design, had been conducted by Morton Thiokol engineers a few months prior, and had shown that the O-rings did not properly seal even at 75 F, a temperature Tufte's chart shows to be perfectly safe for launch.
Based on that experiment, the engineers had been arguing categorically against further launches, at any temperature, in August 1985. Tufte's account makes it sound like the engineers only started putting together some hastily hand-drawn charts at the last minute to stop the launch of the Challenger. But that was only a last effort, and was overruled by managers against the recommendation of the engineers (which was never changed), and despite data that had been known for months.
In addition to ignoring parts of the known history of the catastrophe, Tufte also gets the physics wrong. He focuses solely on O-ring damage, when that was only part of the problem (and not the engineers' main concern), and singles out temperature as the only dependent variable (when the entire design was flawed and temperature only played a partial role). Another one of his criticisms is the inclusion of four test firings of rocket boosters in the data. Those boosters were tied down in a horizontal position when fired, to collect data and to inspect them after they had burned out. Tufte argues that such a configuration does not expose the rockets to the same stresses as an actual launch, without presenting any kind of evidence.
The truth is quite the opposite, though: the higher stresses during a horizontal burn actually qualify a booster for a launch, because the latter is much easier on the rocket. When the rocket is tied down, it vibrates and deforms from the power that has nowhere to go. Also, the horizontal position causes deformation of the round shape, which widens any gaps in the joints.
But all of this is moot in the end, because facts were not the basis for the decision. NASA exerted pressure on Morton Thiokol's management, which approved the launch despite the evidence presented by the engineers.
Conclusions
The paper is very interesting and readable, and makes a great companion to Tufte's book. It tells a cautionary tale about oversimplifying arguments, especially when very serious consequences are involved. Understanding the statistics and visualization is one thing, and there is little doubt that Tufte does that.
What is equally important though, and easily ignored, is that the context of the data also must be understood to create the correct representation. In constructing the case for a simple, elegant chart that seemingly presented all the relevant information and clearly shows the issue, Tufte ignores much of the complexity of the data and physics, and ends up with something that is meaningless.
Wade Robison, Roger Boisjoly, David Hoeker, Stefan Young, Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger. Science and Engineering Ethics, vol. 8, no. 1, pp. 59–81, 2002.
Posted by Robert Kosara on February 2, 2011. Filed under criticism.