Live data from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

seattletimes.com

61–70 of 189 posts

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#61
post #30

Earlier quoted context omitted.

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"

I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#62
> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide.

An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#63
post #33
post #4

Earlier quoted context omitted.

The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…

>Interesting that someone cited divorce and sleep examples as emotional pleas , reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.

I thought a lot of safety thinking was built around acknowledging that humans don't do everything perfectly all the time.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#64
post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

If you view software as a bunch of contracts between different services/libraries/etc. it is hard to ensure that each piece upholds its contract, so we trust them to a degree. I can see how a failure in one place due to load would be hard to catch or gracefully fail from.

That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#65
post #44

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…

> feature of capitalism Software to optimize resource usage while providing a service => blame capitalism is che-guevara-tshirt-edgelord level of inane.

The ad hominem attack isn't appreciated, you need to do better especially if you're going to sincerely defend the virtues of your economic beliefs.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#66
post #33
post #4

Earlier quoted context omitted.

The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…

>Interesting that someone cited divorce and sleep examples as emotional pleas , reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.

Right. Literally every pilot in the US has been taught the IMSAFE mnemonic to assess fitness to fly, even before their first solo in a single-engine piston: Illness, Medication, Stress, Alcohol, Fatigue and Emotion.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#67
post #20

Earlier quoted context omitted.

Both aircraft seat about 180 pax. To be 20,000 lbs off in your hypothesis, each passenger would have to be 110 lbs heavier than modeled, on average. For 30,000 lbs, 165 lbs.

See: https://airinsight.com/the-pending-new-faa-weight-balance-ru... There were recent changes to the weight and balance rules because of increased weight. It's certainly possible this was a factor.

> The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively.

This is a lot smaller in magnitude than +110-165 lb.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#68

Earlier quoted context omitted.

How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…

The SpaceX rockets launch routinely with automation at takeoff and landing. There is no need for human heroic drama in the loop when the software is AAA grade.

> The SpaceX rockets launch routinely with automation at takeoff and landing.

Sure. And there are bugs in their code, too.

https://www.theverge.com/2019/9/3/20847243/spacex-starlink-s...

"SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#69

With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(

How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…

I see this as a near-miss. At my work, there'd be a post-mortem written explaining what went well and what went poorly. Just because not every line of defense was breached doesn't mean everything's peachy.

The first failure was the bug itself - without knowing more about the details it's hard to say more. But one should ask what kind of practices exist at DynamicSource - e.g., compile time thread safety checks, code review practices, testing requirements, rollout procedures - that would prevent bugs like this from getting out into production.

The second failure was that the software failed to validate its own output and fail noisily or generate a warning of some kind. It's safety-critical software - bugs may be excusable, but not giving at least a warning for an unusual output is poor engineering.

The third failure was that the flight staff failed to notice the error until tail strikes happened. Human errors do occur, but we need better systems to assist human operators. E.g., perhaps the flight software itself could act as another line of defense to prompt the pilot when values appear out of normal range.

Yes, in this case the planes weren't anywhere close to crashing. But if the assumption is that bugs are unavoidable, then there needs to be better systems in place to catch those bugs before they cause a major accident. Because the next time we might not get so lucky with detecting the issue and having someone with integrity calling the shots.

Also, no need to start off your comment with an ad hominem - a simple "software errors are unavoidable" would've said just as much.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#70
post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

The key here seems to be the word "many." Alaska Airlines has 289 airplanes (per wikipedia).

All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc...

I just can't see an undue strain being placed on a well designed system from Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...

Post reply on HN