Live data from Hacker News

Failed intercept at Dhahran caused by a software error in handling of timestamps

en.wikipedia.org

41–50 of 91 posts

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#43
post #13

Earlier quoted context omitted.

I learned about it in a Decision Analysis course and had a completely different point driven home. This wasn't bad code. It was code that was correctly written to a very well defined requirement ("System shall be operational for at most X hours before a reboot"). The code was written to a spec that was approved by the customer (the military). Unfortunately though, that requirement wasn't communicated to the end users…

From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to…

The Patriot system was originally designed to operate in Europe against Soviet medium- to high-altitude aircraft and cruise missiles traveling at speeds up to about MACH 2 (1500 mph). To avoid detection it was designed to be mobile and operate for only a few hours at one location.

http://archive.gao.gov/t2pbat6/145960.pdf

Page 2

dug into reference 48 from Wikipedia which referenced this article which I did a search on google.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#44

Earlier quoted context omitted.

I'm failing to find anything that says the requirement was "System shall be operational for at most X hours before a reboot". It's more likely that there was a key performance paramater (KPP) saying that it should be functional for at least some period of time. And that was what was tested. Generally KPPs (which aren't requirements themselves, but influence the requirements for systems) are set at lower bounds, not u…

I just regurgitation about some kind of article the professor brought in. Wikipedia didn't exist when I was taking the course. It's probably in one of the 100 odd source articles since it wasn't just my professor that pointed it out. One of the other commenters mentioned a similar discussion from one of their professors.

Fair. I wasn't replying to you, your #1 sounds a lot like what I'm saying, though.

  1. I remember hearing the system was only designed for
  XX operational hours but was being run over the
  operational spec.
This is very similar to my "at least" which is very different than "at most". In requirements we wouldn't bound ourselves like that. We wouldn't say our system should run for at most 8 hours. We'd say it should run for at least 8 hours. However, we won't say what happens after 8 hours because we don't test it (it's not a requirement). We may communicate to the operators that the system should be rebooted after some period of time if there's a known or anticipated issue, or we may include a soft boot to reset things. For many of our systems, their operating time is usually under 12 hours (they go on aircraft that don't fly for days at a time, mostly), so we never test anything past about 48 hours anyways. If there's an issue that arises around 96 hours, we'd never know from our testing and only know about if an operator pushed it to that limit and recorded the circumstances properly.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#45
post #40

Earlier quoted context omitted.

That's terrible. Competent technical writing is criminally undervalued.

But there's also "presumed" and "did not think" in there. When there's a problem with your killing device you probably shouldn't use it until you've clarified what the problem is and don't just assume your end users will use it correctly. That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag wi…

“You probably shouldn’t use it until you’ve clarified...” doesn’t work so well for a defensive system.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#46
This was a tragic and preventable loss. It's incredible that a software bug might have been the root cause.

At the time, this incident really stuck out because it broke the illusion of our fabled Patriot missile shield protecting us. Civilian expats really believed the inflated Patriot interception rates parroted to us by mainstream media and our American military expat buddies.

A large number of remaining expats who had stuck out the Gulf War to that point decided to pack it in and leave when word got out that the Dhahran barracks were hit. Although history shows that Iraq surrendered days after this incident, at the time there was heightened fear and confusion amongst the remaining expats, especially the non-Americans.

We left on the last Lufthansa flight (crewed by military personnel) after hearing about this.

Nostalgic edit:

During the Gulf War embassies issued equipment and rations to expat citizens who chose to stay behind. Americans were issued full body suits (for adults and youths) due to the biological and chemical weapon payloads that Saddam boasted his SCUDs were carrying, along with MREs that tasted fabulous! In stark contrast, Commonwealth citizens were issued a bare gas mask (adult size only) and mono-flavour MREs that tasted like cardboard.

The British embassy sticks out in my mind: with stern stone-faced expressions they admonished us all for not evacuating and thus endangering children in a war zone. In addition to the terrible rations and gas masks, they wordlessly gave us a stack of translucent stickers. When asked what they were for, embassy staff explained that in the event of the air siren going off, we should get under our sturdiest tables and don our gas masks (standard procedure), and then slap the stickers on. If the stickers changed colour, it meant we were in the presence of a biochemical agent and would have approximately 10 seconds before we died a horrific death.

You kind of had to be there to appreciate the grim humour.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#47
This is bad, editorialized title that is not the title of the article.

Mods should change this. The "software fix" was a software patch which corrected the clocking bug.

The "software workaround" to use pre-fix was reboot.

I hate editorialized, lying titles :(

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#48
I could be reading this wrong, but 1/3 of a second within 100 hours seems really good, like something you'd get from a temperature-controlled crystal oven.

I don't mean to second-guess them in an area I know so little about, but if that was enough to cause a serious issue in the span of only a few days, shouldn't the devices be designed with a separate synchronization system, at least as a backup? Maybe GPS?

Which brings up a sort of interesting question...would a Patriot missile system even have receivers for a weak public signal like GPS, or is it all self-contained?

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#49

The scud missile lead to their deaths, not the software. There's no absolute guarantee it would have intercepted it, plus rebooting a deployed machine regularly is an acceptable fix when it's live in the field

>

Taking 60-90s completely out of protection to reboot a critical defensive system when someone might, at any moment, toss a Mach 5 projectile at you from a couple hundred miles away is a far-from-ideal fix, even if it had been communicated properly to the end users (which it wasn't.)

https://embeddedgurus.com/barr-code/2014/03/lethal-software-...

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#50
post #11

I remember hearing about this in my numerical analysis class. 1. I remember hearing the system was only designed for XX operational hours but was being run over the operational spec. 2. The time was stored in base 10 so the calculation errors added up over time or something like that so if they had used some base 2 timing scheme it would haven't have had issues with rounding errors. My class was in the mid nineties s…

My recollection matches with yours, except I learned about it in the first week of Embedded Systems 101. If it isn't a standard part of the curriculum at every college embedded systems class, it should be! It really drove home the point that bad code can kill.

[deleted]
Post reply on HN