Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

seattletimes.com

Post reply on HN
Print view View original

131–140 of 189 posts

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#131

Post by stateofinquiry »

sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

So much I can't understand here. Why would the load matter- is this a web app? Why would that possibly be a good idea? This seems like software that should run locally, for security, assurance, and auditability reasons.

Follow up thought: When software results are this critical, I wonder if a totally separate program should be used as well and result compared. An independent implementation from another vendor.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#132

Post by Jtsummers »

KennyBlanken wrote: post #126
Jtsummers wrote: post #93

Earlier quoted context omitted.

We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-oper…

In what scenario would you anticipate data for the calculation of rotation speed and engine thrust for one particular plane to be based on a value that is not only shared and writable, but changing each time a plane's performance data is calculated? "multiple data streams feeding in weight data per aircraft"? What does that word salad even mean? Even if what any of what you said was true: these developers are writing…

It is a huge fuckup, I didn't say otherwise.

And that's not word salad, though maybe not as clear as I could have made it. But to clarify for this hypothetical (if this is the issue): Each luggage weigh station submits data to the application or database which then has to tally the weight. Those are multiple data streams, not a hard concept. An airport has many luggage weigh stations all potentially submitting data at the same time.

If those are being processed concurrently and the database or however the data is stored is not properly locked then the tally can be incorrect for the same reason non-atomic increments/decrements can become incorrect in a multithreaded application. But just like the problem with non-atomic (unlocked) increments/decrements the problem may not manifest without high enough load on the system. Low-load testing (what the article seems to describe) means that updates can happen fast enough (server or application is under loaded, less likely to interrupt any update) or submitted slow enough that the issue never appears. So you need to test with a higher load to detect the problem (if you're going to be able detect it in testing at all).

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#133

Post by richwater »

KennyBlanken wrote: post #126
Jtsummers wrote: post #93

Earlier quoted context omitted.

We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-oper…

In what scenario would you anticipate data for the calculation of rotation speed and engine thrust for one particular plane to be based on a value that is not only shared and writable, but changing each time a plane's performance data is calculated? "multiple data streams feeding in weight data per aircraft"? What does that word salad even mean? Even if what any of what you said was true: these developers are writing…

I fucking cracked up at:

"multiple data streams feeding in weight data per aircraft"

it's literally some integers (floats at the worst?).

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#134

Post by jcrawfordor »

phkahler wrote: post #127
mnw21cam wrote: post #116

Earlier quoted context omitted.

> The purpose of telling pilots thrust was to save money and fuel. No, it's more that using lower thrust during take off saves on engine wear and noise levels around the airport. It doesn't really have that big an impact on fuel use, at least that's not the primary purpose. The article concentrates far too much on the thrust setting. The important bit is the speed where the plane should be rotated to take off, which…

It seems like a slower rotation might be useful, to wait until you feel it leave the ground before aiming higher. I'm sure there are reasons for the procedure as is.

Trying to "force the airplane off the ground" by rotating prior to Vr has a few safety downsides. One of them is that it makes tailstrikes more likely, as you'll have to carefully moderate the angle before lift occurs. Another is that it removes loading from the wheels more quickly, which extends the "awkward zone" in which the airplane is still rolling on the ground but has so little weight on the wheels that it will skid very easily. This can lead to aircraft being blown sideways off the runway since you can't yet roll into the wind.

A somewhat less direct concern but a related one is ground effect lift. the aircraft can remain a short distance off the ground (roughly a wingspan of altitude) at lower speeds than it can actually "fly," due to ground effect. Smaller aircraft might routinely spend some time in ground effect gaining additional speed before they begin climbing, but airliners have so much thrust they usually rotate pretty directly to their climb speed. This makes it more of an issue though that if rotation occurs too far before Vy climb speed the ground effect period will be prolonged and increase the amount of time the aircraft spends at risk, flying but slow with poor control authority and too close to the ground to have much of a recovery opportunity. The ground is a pretty safe place to be, the sky is a pretty safe place to be, but that first couple thousand feet between the ground and the sky is rather hazardous and jetliners get out of it as quickly as possible.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#136

Post by justinclift »

ceejayoz wrote: post #27
justinclift wrote: post #14

With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(

How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…

> How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable?

It's literally a safety critical system, not move-fast-and-break-things front end web dev code.

Remember the extraordinarily in depth review of their software while their entire fleet was grounded, where each and every individual bug found was reported in mainstream media?

Where they weren't allowed back in the sky until we were all assured everything was fixed?

The bug they're reporting here sounds like they didn't do reasonable testing. And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems. :/

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#137

Post by basch »

sudhirj wrote: post #125
basch wrote: post #18

Earlier quoted context omitted.

Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.

Not according to the article, the values returned were within the realm of possibility, but wrong. Giving out zeroes might have actually been fine, it’s easy enough to realise there’s a problem if you see your plane weighs 0 pounds.

Hence “some part of a value.”

Such as weight=plane+fuel+people+luggage. If one of the variables became 0 you’d still have the other 3.

It was a low enough value to have made multiple pilots question its value that day, like fuel or passenger load was missing completely.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#138

Post by advisedwang »

sudhirj wrote: post #125
basch wrote: post #18

Earlier quoted context omitted.

Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.

Not according to the article, the values returned were within the realm of possibility, but wrong. Giving out zeroes might have actually been fine, it’s easy enough to realise there’s a problem if you see your plane weighs 0 pounds.

I mean it might be just one part. E.g. luggage gets counted as zero but passenger weight, fuel weight and aircraft weight correct - that might be pluausible but low.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#139

Post by cm2187 »

Aloha wrote: post #2

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think eve…

The bug is in the weight of the payload of the plane. I don't believe pilots have any way to measure the actual weight of the payload of the plane, they rely on that sort of estimates to determine how much fuel they need, take-off speed, etc. It's not really a loss of authority.

Bad weight estimates have resulted in crashes in the past.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#140

Post by ceejayoz »

justinclift wrote: post #136
ceejayoz wrote: post #27

Earlier quoted context omitted.

How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…

> How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? It's literally a safety critical system, not move-fast-and-break-things front end web dev code. Remember the extraordinarily in depth review of their software while their entire fleet was grounded, where each and every individual bug found was reported in mainstream media ? Where they weren't allowed back in…

> It's literally a safety critical system, not move-fast-and-break-things front end web dev code.

So was the Space Shuttle, whose software team is widely regarded as the absolute gold standard in this regard.

https://www.fastcompany.com/28121/they-write-right-stuff

Still, bugs:

"This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."

> Where they weren't allowed back in the sky until we were all assured everything was fixed?

Because no aircraft would fly again with that standard.

> And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems.

The software in question isn't made by Boeing, and it's not just for Boeing aircraft. It's a third-party thing, picked by individual airlines. https://www.dynamicsource.se/ lists support for 13 aircraft across 7 manufacturers.

Post reply on HN