Earlier quoted context omitted.
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
61–70 of 189 posts
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#62An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#63Earlier quoted context omitted.
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…
>Interesting that someone cited divorce and sleep examples as emotional pleas , reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#64> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#65Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this…
> feature of capitalism Software to optimize resource usage while providing a service => blame capitalism is che-guevara-tshirt-edgelord level of inane.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#66Earlier quoted context omitted.
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…
>Interesting that someone cited divorce and sleep examples as emotional pleas , reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#67Earlier quoted context omitted.
Both aircraft seat about 180 pax. To be 20,000 lbs off in your hypothesis, each passenger would have to be 110 lbs heavier than modeled, on average. For 30,000 lbs, 165 lbs.
See: https://airinsight.com/the-pending-new-faa-weight-balance-ru... There were recent changes to the weight and balance rules because of increased weight. It's certainly possible this was a factor.
This is a lot smaller in magnitude than +110-165 lb.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#68Earlier quoted context omitted.
How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…
The SpaceX rockets launch routinely with automation at takeoff and landing. There is no need for human heroic drama in the loop when the software is AAA grade.
Sure. And there are bugs in their code, too.
https://www.theverge.com/2019/9/3/20847243/spacex-starlink-s...
"SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#69With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(
How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to ac…
The first failure was the bug itself - without knowing more about the details it's hard to say more. But one should ask what kind of practices exist at DynamicSource - e.g., compile time thread safety checks, code review practices, testing requirements, rollout procedures - that would prevent bugs like this from getting out into production.
The second failure was that the software failed to validate its own output and fail noisily or generate a warning of some kind. It's safety-critical software - bugs may be excusable, but not giving at least a warning for an unusual output is poor engineering.
The third failure was that the flight staff failed to notice the error until tail strikes happened. Human errors do occur, but we need better systems to assist human operators. E.g., perhaps the flight software itself could act as another line of defense to prompt the pilot when values appear out of normal range.
Yes, in this case the planes weren't anywhere close to crashing. But if the assumption is that bugs are unavoidable, then there needs to be better systems in place to catch those bugs before they cause a major accident. Because the next time we might not get so lucky with detecting the issue and having someone with integrity calling the shots.
Also, no need to start off your comment with an ad hominem - a simple "software errors are unavoidable" would've said just as much.
Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’
#70> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc...
I just can't see an undue strain being placed on a well designed system from Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...