Re-assessing Risk Assessment: A NASA example

Posted by Sten Westgard, MS

An interesting report came out a few days ago. A retrospective internal study of the risks found that "NASA seriously underestimated the dangers astronauts faced when the shuttle fleet began flying in the early 1980s."

Risk study shows NASA underestimated shuttle dangers, Todd Halvorson, Floriday Today, February 13, 2011

"At the time, managers thought there was only a 1-in-100,000 chance of losing a shuttle and its crew. Engineers thought the probability was closer to 1 in 100. But in reality, the odds of a disaster were much higher.

"On each of the shuttle's first nine missions, there was a 1 in 9 chance of a catastrophic accident, according to the new risk analysis. On the next 16 flights that led up to and included the January 1986 Challenger disaster, the odds were 1 in 10."

The full study won't be released until later this year, but even these highlights are sobering. These are rocket scientists applying a very quantitative, rigorous risk analysis, but they still got it wrong. As the difference between the risk perceptions of the managers and engineers illustrates, often the assessment of risk can be influenced by different skills, roles and even motivation. Whenever risk relies on qualitative assessment, there's room for error.

So it's safe to assume that if NASA's best and brightest couldn't get their risk analyses to be completely  accurate, we shouldn't assume that we're going to be able to assess risk perfectly, either. Instead, we should assume that there will always be more residual risk than we realize. That's another sign it's premature to abandon "traditional QC" and reduce our quality control checks to once a week or once a month check.

For a Six Sigma-based Risk Analylsis approach, which uses data-driven quantitative risk assessment, attend the Westgard Workshops 2011.