Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Failed intercept at Dhahran caused by a software error in handling of timestamps

en.wikipedia.org

Post reply on HN
Print view View original

71–80 of 91 posts

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#71

Post by seorphates »

Reboot. Around the same time-frame we gathered the flag for a deployment (fleet admiral) and I was responsible for UNIX systems on the ship. Not long after coming aboard the command came down to reboot all of the systems at midnight, nightly (yes, only the UNIX systems). Being that "But Mister.." never really gets you too far in the military I just rode it iterating through any possible reason for the madness, nightly. I could never come up with a good one. Until now. (ok, perhaps not a "good" reason but crazy enough to count.)

It now makes much more sense to me that a (terrible) mishap had occurred and possible prevention was only a reboot away. I can see how being exposed to that context at upper levels could easily cause one to latch onto any perceived preventative measures.

I also once saw a short ntp time step across multiple clusters (yeh, simultaneously) shut down half of a wafer factory.

Time is important.. but rebooting all your systems at midnight probably will not help you to control it. This especially if there are large, hot, fast objects flying around in the night sky and definitely, really, don't do ALL of them at the same time every day .. especially during, you know, battle. /pro-tip

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#72

Post by dragonwriter »

zeeZ wrote: post #40
macintux wrote: post #30

Earlier quoted context omitted.

That's terrible. Competent technical writing is criminally undervalued.

But there's also "presumed" and "did not think" in there. When there's a problem with your killing device you probably shouldn't use it until you've clarified what the problem is and don't just assume your end users will use it correctly. That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag wi…

> When there's a problem with your killing device

Patriot, in the role deployed in that Gulf War, is a “not being killed” device, but not really a killing device.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#73

Post by Jtsummers »

VLM wrote: post #68
Jtsummers wrote: post #24

Earlier quoted context omitted.

I'm failing to find anything that says the requirement was "System shall be operational for at most X hours before a reboot". It's more likely that there was a key performance paramater (KPP) saying that it should be functional for at least some period of time. And that was what was tested. Generally KPPs (which aren't requirements themselves, but influence the requirements for systems) are set at lower bounds, not u…

Its interesting that FM 44-85 "Patriot Battalion and Battery Operations" is publicly available and pretty easy to find. We discussed this in a systems analysis class back in '04 using a copy of FM 44-85 released in '97. In summary the class blamed TRADOC and the tech writers for publishing a manual that did not accurately reflect real world use cases, with the software bug being a secondary concern. I googled up a co…

The software guys likely never read the field manuals. I know I never did. I read the specs and requirements documents, which are different things than what operators receive. The program office is responsible for maintaining synchronicity between the two with regards to performance parameters (reqs and specs) and performance expectations (manuals). The test office should’ve been familiar with both sides as well.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#74

Post by grkvlt »

leggomylibro wrote: post #48

I could be reading this wrong, but 1/3 of a second within 100 hours seems really good, like something you'd get from a temperature-controlled crystal oven. I don't mean to second-guess them in an area I know so little about, but if that was enough to cause a serious issue in the span of only a few days, shouldn't the devices be designed with a separate synchronization system, at least as a backup? Maybe GPS? Which br…

> weak public signal like GPS

i'd assume it could, since GPS is military, and a mobile missile system is exactly the sort of thing that wants to know where it is, so would have the keys to the (higher resolution) encrypted GPS signals as well.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#75

Post by grkvlt »

leggomylibro wrote: post #60
GCU-Empiricist wrote: post #59

Earlier quoted context omitted.

As a former submariner who has had used clock for inertial navigation or for similar weapons systems 1/3 of a second over 100 hours is terrible.

I mean, it's not an atomic clock, but I'm comparing it to the 32.768KHz RTC crystals I use with consumer microchips. If super-precise isolated accuracy were actually important, I assume they would use a rubidium or cesium oscillator.

1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#76

Post by lostlogin »

mikeash wrote: post #45
zeeZ wrote: post #40

Earlier quoted context omitted.

But there's also "presumed" and "did not think" in there. When there's a problem with your killing device you probably shouldn't use it until you've clarified what the problem is and don't just assume your end users will use it correctly. That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag wi…

“You probably shouldn’t use it until you’ve clarified...” doesn’t work so well for a defensive system.

How long was the off/on cycle? If it was short it would be reasonable to do it periodically. I don't think one can pin the blame on the vendor alone.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#77

Post by lostlogin »

seorphates wrote: post #71

Reboot. Around the same time-frame we gathered the flag for a deployment (fleet admiral) and I was responsible for UNIX systems on the ship. Not long after coming aboard the command came down to reboot all of the systems at midnight, nightly (yes, only the UNIX systems). Being that "But Mister.." never really gets you too far in the military I just rode it iterating through any possible reason for the madness, nightl…

That's still not great logic. Think of all the crazy shit you have seen fix machines. If all the was implemented you would have users doing some truly bizarre things.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#78

Post by mikeash »

lostlogin wrote: post #76
mikeash wrote: post #45

Earlier quoted context omitted.

“You probably shouldn’t use it until you’ve clarified...” doesn’t work so well for a defensive system.

How long was the off/on cycle? If it was short it would be reasonable to do it periodically. I don't think one can pin the blame on the vendor alone.

Clearly the vendor thought it would be reasonable. They failed at communicating it, though. Putting a number on it would have made things clear: “The system must be rebooted after at most 12 hours [or whatever the appropriate value would be] of operation.”

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#79

Post by michaelmrose »

sharemywin wrote: post #43
michaelmrose wrote: post #19

Earlier quoted context omitted.

From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to…

The Patriot system was originally designed to operate in Europe against Soviet medium- to high-altitude aircraft and cruise missiles traveling at speeds up to about MACH 2 (1500 mph). To avoid detection it was designed to be mobile and operate for only a few hours at one location. http://archive.gao.gov/t2pbat6/145960.pdf Page 2 dug into reference 48 from Wikipedia which referenced this article which I did a search o…

The fact that the bug manifests after a longer than normal period of operation doesn't ex post facto make it not a bug. If you add 2 and 2 and get 42 you failed.

It is however a good explanation why it remained undetected.

Re: Failed intercept at Dhahran caused by a software error in handling of timestamps

#80

Post by michaelmrose »

Dehstil wrote: post #26
michaelmrose wrote: post #19

Earlier quoted context omitted.

From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to…

If by "defective" you mean has rounding errors, then sure. Everything that rounds numbers is defective. To be fair, round errors can sometimes be mitigated by carefully changing the order of operations, but never fully eliminated in those cases.

You can avoid rounding errors 100% of the time for as long as you like. For example you can use integers.

Its entirely possible to have any reasonable degree of precision reasonably required to the limits of our tools to measure.

This isn't about an inherent limit of computation its just programmer error.

Post reply on HN