Tufte and the Truth about the Challenger

Almost exactly 25 years ago, on January 28, 1986, Space Shuttle Challenger disintegrated seconds after lift-off. One of Edward Tufte’s most famous examples of bad charts are the ones used by engineers who argued against the launch, and who failed to convince. It’s a fascinating story, but it has one major fault: it is not true.

The Charge

While it is easy to look at this as purely a question of proper data display, the claim Tufte makes is a very serious one: the engineers at Morton Thiokol, which manufactured the rocket boosters that led to the disaster, failed to communicate their concerns, and were thus partially responsible for the death of seven astronauts.

The authors of the paper this posting is based on, Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger (details below), claim that Tufte both misunderstood and misrepresented the argument and the true cause of the failure. What is more, Tufte keeps making the case that it was the engineers’s fault, despite the fact that information is readily available that shows what really happened.

O-Ring Failure

To briefly summarize the issue, what happened was that the aft joint on one of the Challenger’s rocket boosters failed during launch, which led to the escape of hot gases to the side, towards the main fuel tank. That tank was quickly ruptured, and started to disintegrate. The tank and shuttle were then torn apart by aerodynamic forces (the Wikipedia article argues that there was no actual explosion).

Tufte has argued that the main cause for the joint failure was damage to the O-rings, which sealed the joint to prevent gases from escaping through a gap between parts of the rocket. That failure was supposedly due to the fact that cold temperatures (colder than any shuttle launch before) had led to a loss of flexibility in the rubber-like material the O-rings were made from.

The main argument in the paper is that the cause of the failure in the joint was not in fact O-ring damage, but blow-by: hot gases passing by the O-rings when they do not properly seal. Also, Tufte’s elegant chart showing temperature versus damage is misleading, because it does not show the actual O-ring temperature, but a mix of O-ring and environmental temperature (which can differ significantly).

Physics

Tufte describes Richard Feynman’s famous demonstration of the O-ring’s lack of flexibility after having been compressed in a glass of ice water. He criticizes the lack of a control on this experiment, and that Feynman did not make clear what he was comparing (temperature, wet conditions, etc.), and to what reference. But a very similar experiment, with controls and with a much better design, had been conducted by Morton Thiokol engineers a few months prior, and had shown that the O-rings did not properly seal even at 75 F, a temperature Tufte’s chart shows to be perfectly safe for launch.

Based on that experiment, the engineers had been arguing categorically against further launches, at any temperature, in August 1985. Tufte’s account makes it sound like the engineers only started putting together some hastily hand-drawn charts at the last minute to stop the launch of the Challenger. But that was only a last effort, and was overruled by managers against the recommendation of the engineers (which was never changed), and despite data that had been known for months.

In addition to ignoring parts of the known history of the catastrophe, Tufte also gets the physics wrong. He focuses solely on O-ring damage, when that was only part of the problem (and not the engineers’ main concern), and singles out temperature as the only dependent variable (when the entire design was flawed and temperature only played a partial role). Another one of his criticisms is the inclusion of four test firings of rocket boosters in the data. Those boosters were tied down in a horizontal position when fired, to collect data and to inspect them after they had burned out. Tufte argues that such a configuration does not expose the rockets to the same stresses as an actual launch, without presenting any kind of evidence.

The truth is quite the opposite, though: the higher stresses during a horizontal burn actually qualify a booster for a launch, because the latter is much easier on the rocket. When the rocket is tied down, it vibrates and deforms from the power that has nowhere to go. Also, the horizontal position causes deformation of the round shape, which widens any gaps in the joints.

But all of this is moot in the end, because facts were not the basis for the decision. NASA exerted pressure on Morton Thiokol’s management, which approved the launch despite the evidence presented by the engineers.

Conclusions

The paper is very interesting and readable, and makes a great companion to Tufte’s book. It tells a cautionary tale about oversimplifying arguments, especially when very serious consequences are involved. Understanding the statistics and visualization is one thing, and there is little doubt that Tufte does that.

What is equally important though, and easily ignored, is that the context of the data also must be understood to create the correct representation. In constructing the case for a simple, elegant chart that seemingly presented all the relevant information and clearly shows the issue, Tufte ignores much of the complexity of the data and physics, and ends up with something that is meaningless.


Wade Robison, Roger Boisjoly, David Hoeker, Stefan Young, Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger. Science and Engineering Ethics, vol. 8, no. 1, pp. 59–81, 2002.

Comments

  1. paresh shah says

    “What is equally important though, and easily ignored, is that the context of the data also must be understood to create the correct representation” – an important lesson that!

  2. Claire says

    Thanks for posting this–it reminds me quite a bit of how, when I first started reporting, I’d create stories first, and then add in only the quotes that supplanted them. I’ve a feeling Tufte was setting out to prove that the Morton Thiokol Engineers had erred, and was intent on showcasing only evidence that helped his theory.

  3. Ellie Kesselman says

    I read the journal article, well, skimmed it. Published in 2002. But that doesn’t detract at all. What an excellent find!

    Tufte’s error is due to his perspective (and perhaps arrogance). With historical distance, it is far easier to find fault. Let me rephrase that and use an example: Tufte’s analysis of Napoleon’s failed attempt to conquer Russia was flawless. But his analysis of the Challenger disaster shows how poorly suited Tufte is to the nitty gritty of incident review. The latter requires the ability to empathize and re-enact the situation, as one of the participants. Apparently that is not one of Tufte’s abilities.

    I guess Tufte isn’t perfect after all! That’s okay: one needn’t be able to do everything. But one also should be aware of one’s limitations….

    Thank you so much for a wonderful article and critique!

  4. ColdWarVet says

    I was sent to a Tufte class many years ago as a requirements analyst with an organization that developed the On-board Shuttle flight software.

    Tufte’s depiction of the temperature versus o-ring damage instances was jaw dropping to me. I showed it to many folks in an effort to convey the need to better depict Avionics and Software metrics and test results.

    Tufte will be in Houston for an 08Apr11 course and the email calls for students are going out now. Though the course is thoroughly engaging and delivers useful information, this article and referenced paper should be mandatory reading for anyone planning to attend.

    • Johnny says

      Are you kidding? Tufte doesn’t even let his own students ask him any questions. I saw his lecture in Boston and was aghast when he flippantly dismissed a valid question from a student with a wave of his hand. He’s a legend in his own mind and is worshiped like a god by the legions of corporate do-nothings who get sent to his lectures by management. A critical eye can easily see right through this fraudster.

  5. Robert Kosara says

    I thought of that too, but I don’t think it’s a big issue. Boisjoly isn’t just criticizing Tufte, he has also been very outspoken about the processes and decision-making structures that led to the launch decision. Plus, he has first-hand knowledge of what the engineers were doing and thinking, and what they were trying to accomplish. So his view is very valuable in figuring out what really happened.

  6. Robert Kosara says

    I did searches on Tufte’s website for the names of the authors, but they came up empty. I can’t believe that nobody has asked him, though. It’s not even about “recent,” the paper is from 2002. Since he tightly controls what gets posted on his “forum,” I guess that he simply doesn’t want to talk about it.

  7. Anonymous says

    I attended Tufte’s course based on a colleague’s recommendation (2 years ago). At the time, I thought some of his comments during the course was over my head (I thought maybe I was not at this level to understand it). He is definitely an IDOL but he gives himself too much credit at times.

  8. Lord John Whorfin says

    He’s essentially accusing the engineers of incompetence when in reality he’s an arrogant windbag full of himself. I never had much respect for his ridiculous books and his utterly insane demands on print quality, but this exposure of his utter incompetence and lack of insight is priceless. What a putz.

  9. Anonymous says

    can you please post or ask those who just attended Tufte’s course in Houston to share their stories with the group?

  10. Ellie Kesselman says

    You expressed more clearly what I glossed over in my comment. Tufte’s attitude toward the engineers, that they were incompetent, that bothered me too. Rankled. Seemed very arrogant.

    @EagerEyes this was a good post. Your entire blog is impressive. I find myself returning to it, re-reading the same posts, again and again!

  11. Kyle Hailey says

    “It’s a fascinating story, but it has one major fault: it is not true.”
    That’s an exaggeration. A greater exaggeration than Tufte’s original story. Tufte glosses over some details like whether or not the engineers had the ambient temperatures of flights without damage and his analysis is overly confident and cocky but he’s trying to demonstrate a crucial point. That point is good visualization of data is powerful and even in these modern times that point is often lost. When I designed the performance monitoring interface to Oracle’s Enterprise Manager in 2003 I spent about 6 months argue for visualization of data while kernel developers argued for just textual data. The story Tufte tells, as I see it, is about engineers were concerned with the “if”, “and” and “but” of the situation because there were questions such as why did the desert trials not have damage in the 40 degree range while the flight had major damage at 53. The answers were not black and white, They did have a black and white opinion though which was there should be no flights below 53 degrees and that colder temperatures posed more of a threat.
    I see the story as a powerful tragedy that shows the break down in communication between management and engineers. Engineers try to explain everything. When there is too much information management doesn’t get it and makes a misinformed decision. I see it all the time. Engineers are not understood in textual performance reports showing the root cause and clearly pointing to the solution path. Manager’s eyes glass over and they continue to hold meetings and preside over debates when the answers are there. On the other hand when data is presented graphically managers click in and take action. Here is my take on the data

    The correlation between damage and temperature is unclear but it is compelling. There is no damage above 75. There is always damage below 65. The incidents of damage rise below 65. But my favorite part, and the part the is compelling graphically, is that there is no data below 53 degrees and the flight was going to be at around 29. Eve with the ground testing data was well above 29 degrees. How could they launch when there were clear concerns about damage and temperature due and there was no data below the mid 40 degrees? Sure the solid rocket joint temperatures might be different than ambient but the would be with in a few degrees. The damage at 75 was different than the damage at other temperatures, thus it is colored differently. At 70 there were many successful launches and 2 with damage. The middle black dot is the normalized damage ie average of success and damage. The two grey dots represent the un-normalized data for 70.
    Tufte graphic revisited
    The most unfortunate part of Tufte’s story is any blame on the engineers. If anyone is to blame it seems to be management who didn’t fully take the time to understand the engineers objections. I think the main thrust of the study above is that Tufte cut too many corners in his analysis and wrongly blames the engineers when the situation was much more complex. I agree with that. I also agree that the graphics, had they been an option, which the weren’t at the time, would have been a powerful addition.

  12. Royce Michael Lee says

    The authors of the Tufte rebuttal appear to have erred — the temperatures in Tufte’s table refer to O-ring temperatures only and seem to me to exactly match those appearing on the Morton Thiokol infographic presented to the Presidential Commission on the Space Shuttle Challenger Accident that Tufte’s book references (and also appears here). All other qualitative references to temperature in Tufte’s analysis seem to refer to O-ring temperatures exclusively. The only mistake I can find in Tufte’s scatterplot is the predicted O-ring temperatures for Challenger (SRM 25), which the engineers had estimated would be 27-29 degrees, were shown on the Tufte chart as 26-29 degrees. Otherwise the scatterplot and its labels appear consistent with the data.

    Another issue I have with the Robison, Boisjoly, Hoecker and Young paper is footnote g on page 65. This footnote discusses the confusion over flights and rocket names. To ‘clear-up’ the confusion between Morton Thiokol’s naming scheme and Nasa’s naming scheme, the authors create a 3rd scheme of their own. This scheme is confusing and redundant: confusing in that the prefix they chose (STS) is the same prefix that NASA has used; and redundant in that the numbering scheme they use matches Morton Thiokol’s (numbered 1 through 25). Simply sticking to Morton Thiokol’s prefix and numbering (e.g. SRM 25 for the Challenger disaster) as Tufte did would have aided clarity.

  13. Mokeman says

    I’ve read all of the papers, about who did, and said what when, but I have unanswered questions.

    Regardless of how much pressure (future contract threats and the like) by Malloy and NASA would have placed on MTI’s Lund, Mason, Kilminster, and Wigley, what possibly would have possessed them (MTI’s Lund, Mason, Limminster etc.) to ultimately sign, in writing, ANY rationale for “recommending” a launch at 27-29 degrees F when the the SRB itself and it’s components were only “qualified” for 40-90 def F ?? I mean, if well-known, documented SPEC’s were for 40-90 deg., what business did any of the parties even have to the night before conference call in the first place?

    And that’s not to even mention the known issues, and 5-7 flights with signs of blow by. Given the known situation, the phone calls/conference with MTI should have only occurred if launch temperature was predicted to be ABOVE 40 degrees, and *below* 53 degrees (the coldest previous launch).

    Tufte talks a whole bunch about the role of the engineers and such, but all of that goes out the window the second that Kilminster called for the 5-min recess in the conference (that actually lasted 30 minutes). As soon as the 4 managers over-ruled the engineers, it was all a moot point in my mind.

    Since Tufte goes on to talk about how the case could have been “better presented”, it wasn’t clear to me who he was implying that the case could have been presented better TO, – the MTI managers, or Malloy/NASA? Remember, the MTI engineers (Kilminster) are already been previously been convinced by their engineers that a launch was NOT recommended, and conveyed that quote clearly to all parties.

    Regardless of pressure, he (Kilminister specifically) could have simply hid behind basic contractual/legal SPEC’s, and hand the hot potato issue back up to Mason and Lund, and simply say: here boss, – YOU explain to the customer WHY anyone here at MTI would “sign” a piece of paper stating that a launch is recommended far outside of contract/qualified SPECS. Period, end of story. What did I miss here?

Leave a Reply