Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

seattletimes.com

Post reply on HN
Print view View original

41–50 of 189 posts

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#41

Post by hn_throwaway_99 »

lupire wrote: post #32
hn_throwaway_99 wrote: post #26

Earlier quoted context omitted.

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to…

"huge" is not the same as "horrible". One racy boi can take out an airplane.

But calling something an "internal architectural problem" implies things are fundamentally designed wrong, and the code needs a rewrite. My point was the architecture can definitely be fine, but you can still have bugs (even catastrophic ones) because someone left out a `synchronized` keyword somewhere.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#42

Post by Ozzie_osman »

hn_throwaway_99 wrote: post #26
sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to…

True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof".

If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#43

Post by terr-dav »

Game_Ender wrote: post #23
1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

In a competitive industry saving money can mean lower fares for customers. It can also mean fewer total planes or engines produced reducing environmental side effects of that activity.

It can mean those things, yes, but only as side-effects. The primary effect being to increase shareholder value.

There’s no need to have market based competition to keep costs down. Look at USPS for example. That operation is run without a profit motive, serves literally every zip code in the US (by definition), and runs healthily year after year (except for the part where Congress imposed arbitrary pension funding requirements[1] and private capital interests have been chomping at the bit to buy up all the real-estate holdings).

They also don’t have UPS’s problem of not putting AC in their delivery vehicles.

[1] https://ips-dc.org/how-congress-manufactured-a-postal-crisis...

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#44

Post by pxmpxm »

1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

> feature of capitalism

Software to optimize resource usage while providing a service => blame capitalism is che-guevara-tshirt-edgelord level of inane.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#45

Post by 1101010010 »

pixl97 wrote: post #28
1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

I mean yes, would you rather flights are $10,000 or $100,000 each so the engines are taken apart and rebuilt each time after a flight? Reducing the maintenance interval requirements 'safely' is good for everyone.

Let's reduce the intervals even further so flights are only $10 then, right?

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#46

Post by sidlls »

lupire wrote: post #35
Aloha wrote: post #12

Earlier quoted context omitted.

They may have thought their test coverage, covered it adequately - who knows, the success was on Alaska's part. Also process, procedure and more importantly people are always the last line of defense against failure.

Making excuses for the failures in the first line of defense is the first domino in catastrophic failure. Normalization of deviance.

I don't think OP was making excuses. It is true that the software vendor's missing test case indicates a poor quality development environment and that humans are the last line of defense. No development process is perfectly resilient against errors, even good quality processes.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#47

Post by exabrial »

sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

I was just about to post the _Exact_ same thing. Each execution of the program should be completely independent from other calculations. Something is _Horribly_ wrong with their architecture.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#48

Post by jcutrell »

Seems like something as critical as thrust in this phase of flight should have some kind of cross-check validation. Sounds like a bad single point of failure to me, but maybe there is a cross check I don’t know about that isn’t covered in the story.

What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#49

Post by hn_throwaway_99 »

Ozzie_osman wrote: post #42
hn_throwaway_99 wrote: post #26

Earlier quoted context omitted.

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to…

True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof". If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).

I'm not "brushing mistakes off", at all. The process that led to the bugs needs a full review and root-cause corrective action, especially enhanced testing scenarios. I'm just pointing out that software "being wrong under load" is not some sort of crazy, unusual problem that points to poor software architecture. These bugs can easily be caused by a single line of code that wasn't synchronized correctly.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#50

Post by jcutrell »

fwlr wrote: post #17

“the bug was missed because it only presented when many aircraft at the same time were using the system.” The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load? The explanation that comes to mind is that DynamicSource has a subservice for each sou…

Makes me think this could have been a race condition where these aircraft somehow received the load calculations from other aircraft being calculated at the same time window. Some kind of poorly managed event stream / consumer that assumed no parallel jobs would be run? Or maybe the jobs are deduped by departure / arrival instead of flight number? Seems possible
Post reply on HN