This reminded me of the problem of unit testing vs integration testing. Sometimes, no matter how much code coverage you have, the unit tests don't find that critical bug that takes everything down. Just like testing the 2 square feet of foam didn't find the problem. You also need integration testing.
How We Nearly Lost the Discovery Shuttle
21–30 of 63 posts
Re: How We Nearly Lost the Discovery Shuttle
#22"We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..." That to me is pretty disgusting. In an incident like the loss of Columbia, there is no one, true "root cause". To assign blame to those foam technicians was disingenuous and just another instance of "passing the buck" that seems to happen so often in the post-mortem of NASA failures. NASA knew of e…
The problem with the investigation into foam loss on STS-107 was that when they tested small sections of foam, it was free of material defects. At the time, investigators concluded incorrectly, by exclusion, that it must have been faulty installation that caused the loss of the shuttle. NASA management blamed those workers for the loss of Columbia. When Discovery's External Tank had cracks and found it was cracking from thermal stresses. This meant that the blaming it on faulty installation was wrong, and for which Wayne Hale apologized.
Re: How We Nearly Lost the Discovery Shuttle
#23I loved this story. This is a good example of how you can go down the rabbit hole of solving a particular problem without stepping back to consider if the problem you are solving is key to getting the result you want.
It's amazing to hear someone be so honest about this.
Re: How We Nearly Lost the Discovery Shuttle
#24This guy must need antacid like nobody's business. In some ways I envy him a little... I always try to assign more importance to my job than is really warranted; he has no call for that.
Re: How We Nearly Lost the Discovery Shuttle
#25"We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..." That to me is pretty disgusting. In an incident like the loss of Columbia, there is no one, true "root cause". To assign blame to those foam technicians was disingenuous and just another instance of "passing the buck" that seems to happen so often in the post-mortem of NASA failures. NASA knew of e…
I'm just impressed he tells the story so honestly. He could have characterized what was said differently.
And I wonder if there is a bias in his telling... that he words it harsher than it was presented in reality out of a sense of guilt.
Re: How We Nearly Lost the Discovery Shuttle
#26Earlier quoted context omitted.
In that, the Shuttle was very much like the Concorde: a unique and complex system beyond the edge of knowledge (at its creation), full of flaws and working through a combination of sheer luck and heroic efforts.
Actually the space shuttle is the safest launch vehicle to date. From the ones that have at least 100 launches (in order to be able to properly compute stats for them), here are their failure rates as taken from http://www.ontonix.com/Blog/Outliers_-_understanding_Nature_... : 2% US Space Shuttle 5% R-7 (Russian Soyuz) 5% Ariane 1-4 (European) 6% Tsyklon (Russian) 7% Kosmos (Russian) 10% Thor/Delta/N1/N2/H1 (US) 11%…
I was involved in a similar discussion a couple of weeks ago. I was looking at the same figures you were and complimented the space shuttle because of it.
I was wrong, and you're making the mistake that I did. Namely confusing reliability and safety. A reliable rocket is one that successfully does what it is supposed to. A safe rocket is one that doesn't kill people.
The US space shuttle has proven to be more reliable than the Soyuz. It is more likely to actually get you into space. But the Soyuz has been safer than the US space shuttle. If you try to get into space on it, you're less likely to die.
If this seems impossible, consider that in both Soyuz 18a in 1975 and Soyuz T-10-1 in 1983 the rocket failed, but the cosmonauts survived. (In the first case the rocket failure happened 90 miles in the air, but the cosmonauts survived.) The space shuttle, by contrast, had no successful aborts.
Re: How We Nearly Lost the Discovery Shuttle
#27"We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..." That to me is pretty disgusting. In an incident like the loss of Columbia, there is no one, true "root cause". To assign blame to those foam technicians was disingenuous and just another instance of "passing the buck" that seems to happen so often in the post-mortem of NASA failures. NASA knew of e…
As additional color here is what he posted in a reply to a reply of his blog entry:
Perhaps I was a little too brief in my writeup. Nobody that I know (certainly not me) went to MAF and downdressed any of the workers. What happened was that the conclusion was reached in engineering and management meetings and the word filtered out to the workers that poor workmanship was the proximate cause of the loss of Columbia – as if there wasn’t enough blame to go around in many other areas. I really regret the erroneous conclusion, the impact it made on the workers, and the way the whole scenario played out at MAF. The people there were very hard working, dedicated, and proud of their involvement with America’s space program, many of them second or third generation workers at that location. Now, of course, they have all been laid off and the MAF plant is virtually a ghost town with very limited work going on there for other NASA or commercial projects.
Re: How We Nearly Lost the Discovery Shuttle
#28"We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia..." That to me is pretty disgusting. In an incident like the loss of Columbia, there is no one, true "root cause". To assign blame to those foam technicians was disingenuous and just another instance of "passing the buck" that seems to happen so often in the post-mortem of NASA failures. NASA knew of e…
I'm just impressed he tells the story so honestly. He could have characterized what was said differently.
Yes, I recoiled when he said the line quoted above.
But I think this direct wording was a rhetorical device to set the hook for the pivot in the story, when it turned out the problem with the foam wasn't inclusions during installation at all.
Re: How We Nearly Lost the Discovery Shuttle
#29A great post, especially since it seeks to get at the truth of something that has implications for future missions, at the risk of the OP's reputation. This part is one of the more disturbing parts though, and a good reminder of why technical persons of all fields, whether rocket scientists or programmers, should not adopt a "Well, we worked hard and we're smart so I'm sure everything's fixed" > What you probably don…
No, it's about uncertainty.
You have a stated problem "the foam that came off didn't come off because of the reasons we thought it did." Now you have no other ideas besides what you've already considered and tested for 26 months. What do you do? Possibly spend another 2 years investigating and find nothing? Or conclude that the risk is small enough to fly while being vigilant about the problem and looking for more data to lead you in the right direction?
Sometimes the only way to get more data to solve the problem is to do the very thing that causes it, while hoping that you've mitigated its effects well enough that the system is still safe.
Re: How We Nearly Lost the Discovery Shuttle
#30A great post, especially since it seeks to get at the truth of something that has implications for future missions, at the risk of the OP's reputation. This part is one of the more disturbing parts though, and a good reminder of why technical persons of all fields, whether rocket scientists or programmers, should not adopt a "Well, we worked hard and we're smart so I'm sure everything's fixed" > What you probably don…
It's hard.
Despite all the "if it's not safe, say so" posters (e.g., http://www.dpvintageposters.com/cgi-local/detail.cgi?d=9203), the anonymous tip lines, and everything else, it's hard to stand up and say that something is not safe enough, or that this cause has not been fully nailed down. Because it's usually a qualitative thing, and careers and programs are at stake.
I was at a large auditorium at JSC (Houston) once. It's where the big pre-launch briefings are held. They had installed phone handsets all over the periphery and aisles of the room so that anyone could easily stop a briefing to ask a question. (I've never seen a capability quite like that in an auditorium.)
The room had (IIRC) around 200 seats. It's hard to be the guy who stands up and stops the briefing to ask the key question. Even though a lot of infrastructure has been created to make it possible.