Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

seattletimes.com

Post reply on HN
Print view View original

31–40 of 189 posts

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#31

Post by CPLX »

guitarbill wrote: post #19
hodgesrm wrote: post #5

That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.

Agreed, although it takes an individual with a huge amount of integrity or a culture that values and promotes integrity to make a call like this: > “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.” Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing w…

Amazingly, this really is the approach that has characterized all aspects of aviation for decades. It’s incredibly admirable and worth emulating in many other situations.

It’s also fragile, and the fact that it strikes many as an uncommon level of integrity for a business decision is precisely why aviation needs to be shielded from having the prevailing business culture sneak in, like we have seen happen at Boeing in a few highly notable instances.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#32

Post by lupire »

hn_throwaway_99 wrote: post #26
sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to…

"huge" is not the same as "horrible". One racy boi can take out an airplane.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#33

Post by pxmpxm »

basch wrote: post #4
Aloha wrote: post #2

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think eve…

The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…

>Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human.

Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#34

Post by almenon »

1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

Less fuel is better for the environment. That should be counterbalanced with safety, which they did - even with a bug and a heavy plane the plane still took off safely.

This means they are also leaving a safety margin of fuel, even though that costs extra money.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#35

Post by lupire »

Aloha wrote: post #12
sidlls wrote: post #7

Earlier quoted context omitted.

This is a failure in the development of the software that good process in the purchaser’s handbook saved them from. That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.

They may have thought their test coverage, covered it adequately - who knows, the success was on Alaska's part. Also process, procedure and more importantly people are always the last line of defense against failure.

Making excuses for the failures in the first line of defense is the first domino in catastrophic failure. Normalization of deviance.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#36

Post by hn_throwaway_99 »

djanogo wrote: post #30
cratermoon wrote: post #15

Earlier quoted context omitted.

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"

There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare....

Obviously sessions should be independent and not sharing data, but that's why it was a bug.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#37

Post by kfajdsl »

1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

I don't see how this is an example of that? Saving fuel and reducing wear on the parts is a good thing. In this case, nobody's lives were in danger either.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#38

Post by CPLX »

1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

That’s a feature of engineering. Figuring out what performance and safety requirements are and then optimizing the resources required to meet those requirements is what engineering is for.

The influence of capitalism would be on the requirements. But assuming the requirements are well-considered trying to minimize fuel and engine usage to meet those requirements is a good thing.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#39

Post by brookst »

1101010010 wrote: post #21

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

You think it would be more ideal to wear out engines sooner and burn more fossil fuels?

If efficiency is the curse capitalism brings us, I’m feeling a little better about it.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#40

Post by johntb86 »

Tempest1981 wrote: post #11

Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system > the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings. Multithr…

> Multithreading/contention issue? But how would that alter the weights?

Maybe they stored something in a global variable, and with two planes querying at the same time values for the wrong plane could be retrieved.

Post reply on HN