Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Failed intercept at Dhahran caused by a software error in handling of timestamps

en.wikipedia.org

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#81

Post by michaelmrose »

Jtsummers wrote: post #24
pilom wrote: post #13

Earlier quoted context omitted.

I learned about it in a Decision Analysis course and had a completely different point driven home. This wasn't bad code. It was code that was correctly written to a very well defined requirement ("System shall be operational for at most X hours before a reboot"). The code was written to a spec that was approved by the customer (the military). Unfortunately though, that requirement wasn't communicated to the end users…

I'm failing to find anything that says the requirement was "System shall be operational for at most X hours before a reboot". It's more likely that there was a key performance paramater (KPP) saying that it should be functional for at least some period of time. And that was what was tested. Generally KPPs (which aren't requirements themselves, but influence the requirements for systems) are set at lower bounds, not u…

Getting a slightly wrong answer ought to have been detectable even after short period of time even if the difference was microscopic.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#82

Post by Jtsummers »

michaelmrose wrote: post #81
Jtsummers wrote: post #24

Earlier quoted context omitted.

I'm failing to find anything that says the requirement was "System shall be operational for at most X hours before a reboot". It's more likely that there was a key performance paramater (KPP) saying that it should be functional for at least some period of time. And that was what was tested. Generally KPPs (which aren't requirements themselves, but influence the requirements for systems) are set at lower bounds, not u…

Getting a slightly wrong answer ought to have been detectable even after short period of time even if the difference was microscopic.

Not the way we test these things. You set the KPPs and analyze system performance. Especially back then, there wouldn’t have been much in the way of unit testing or anything for these sorts of systems.

You set your performance parameters (have some success rate while operating continuously for up to 4 hours). Then you launch missiles at it (simulated and real). If you stop enough of then you’re good.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#83

Post by michaelmrose »

Jtsummers wrote: post #82
michaelmrose wrote: post #81

Earlier quoted context omitted.

Getting a slightly wrong answer ought to have been detectable even after short period of time even if the difference was microscopic.

Not the way we test these things. You set the KPPs and analyze system performance. Especially back then, there wouldn’t have been much in the way of unit testing or anything for these sorts of systems. You set your performance parameters (have some success rate while operating continuously for up to 4 hours). Then you launch missiles at it (simulated and real). If you stop enough of then you’re good.

Article discussing testing software back in 1976

https://dl.acm.org/citation.cfm?id=807721

No real good excuse for not actually testing systems that can take or save lives.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#84

Post by mikekchar »

michaelmrose wrote: post #19
pilom wrote: post #13

Earlier quoted context omitted.

I learned about it in a Decision Analysis course and had a completely different point driven home. This wasn't bad code. It was code that was correctly written to a very well defined requirement ("System shall be operational for at most X hours before a reboot"). The code was written to a spec that was approved by the customer (the military). Unfortunately though, that requirement wasn't communicated to the end users…

From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to…

Conversations like this are surprisingly common in our industry ;-) To help ease communication there are 2 terms in common usage: software error and bug. A software error is code that is incorrect. A bug is a software error that manifests a user visible problem. In this case the incorrect code is a software error, but it does not manifest a user visible problem unless it is used outside some assumed parameters. The bug doesn't exist when the product is used as intended. One can argue that the behaviour is undefined when used outside of the intended use and therefore there is no bug. There is no arguing about the software error, though. It exists.

Arguing about whether or not something is a bug is pointless precisely because someone will just pull the "behaviour outside of expected use is undefined" thing out of the bag. Regardless of whether or not you should have expected something to work, if your product unintentionally kills people due to a software error, you have a gigantic problem. It's really that lesson we have to keep in mind.

I get this all the time from project managers: it doesn't matter if X fails because we aren't designing the software for X. But you can't just dismiss X -- you need to understand the consequences of X just in case somebody tries to do it. For example: It corrupts the DB if 2 people edit the same record at the same time. The project manager says, "Not a problem. I got sign off from the groups using the app and they promise never to have 2 people working on the same thing. Problem solved, and no need to modify the code!" Of course a week later the DB is corrupted and it's not a bug (it's a feature ;-) ).

It does make software development more costly, and you need to draw the line somewhere. This requires balancing risk. But I will argue that if you are writing software for a missile, there is no hiding behind the "we didn't design it for that" argument.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#85

Post by cocoablazing »

grkvlt wrote: post #75
leggomylibro wrote: post #60

Earlier quoted context omitted.

I mean, it's not an atomic clock, but I'm comparing it to the 32.768KHz RTC crystals I use with consumer microchips. If super-precise isolated accuracy were actually important, I assume they would use a rubidium or cesium oscillator.

1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?

USN ballistic missile submarines deploy the most accurate INS in the world (ESGN), and that system is used in conjunction with another advanced gyro.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#86

Post by muthdra »

sli wrote: post #8
nathan_long wrote: post #7

Earlier quoted context omitted.

Oh, gosh. Really? It makes me kind of sick imagining that call.

You're not even curious to see how it was dealt with and how the issue was expressed to the vendor? I'd never be in a meeting regarding deaths of users of my software, because I just make internal webapps, so I just cannot help but be curious as to how one of those meetings would go.

I'm not, because it would bore me. I see shit like that for breakfast when studying transportation. But if you don't do it, I say it's because of your own primal instincts, so you stuff them down...

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#87

Post by girvo »

otoburb wrote: post #46

This was a tragic and preventable loss. It's incredible that a software bug might have been the root cause. At the time, this incident really stuck out because it broke the illusion of our fabled Patriot missile shield protecting us. Civilian expats really believed the inflated Patriot interception rates parroted to us by mainstream media and our American military expat buddies. A large number of remaining expats who…

That description of the stickers is so British it makes me think of Monty Python. Brilliant!

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#88

Post by GCU-Empiricist »

cocoablazing wrote: post #85
grkvlt wrote: post #75

Earlier quoted context omitted.

1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?

USN ballistic missile submarines deploy the most accurate INS in the world (ESGN), and that system is used in conjunction with another advanced gyro.

Here is a 2 RU atomic clock on ebay for context: https://www.ebay.com/itm/Antelope-Audio-Isochrone-10M-Rubidi...

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#89

Post by godelmachine »

avar wrote: post #3

Even better, the timeline: - February 11th: Vendor informed of the issue - February 25th: 28 people die because of the issue - February 26th: The vendor ships a fix I'd have loved to be a fly on the wall for that phonecall on the 25th (or early on the 26th).

Who was the vendor?

Aren't defense contractors required to be on their toes all the time?

EDIT → Found it. The PATRIOT Project Office.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#90

Post by grkvlt »

cocoablazing wrote: post #85
grkvlt wrote: post #75

Earlier quoted context omitted.

1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?

USN ballistic missile submarines deploy the most accurate INS in the world (ESGN), and that system is used in conjunction with another advanced gyro.

Not sure about that. ESGN is old technology, since submarines don't need that much accuracy. For example, space probes, ballistic missiles, smart artillery shells/rockets/missiles and so on would all appear to have multiple orders of magnitude better accuracy than submarines, in the fractional ppb ranges, rather than tens of ppm. [0][1]

0. https://www.sto.nato.int/publications/STO%20Meeting%20Procee...

1. http://users.cecs.anu.edu.au/~Jonghyuk.Kim/teaching/Inertial...

Post reply on HN