121–130 of 189 posts
Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?
“the bug was missed because it only presented when many aircraft at the same time were using the system.” The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load? The explanation that comes to mind is that DynamicSource has a subservice for each sou…
Earlier quoted context omitted.
The key here seems to be the word "many." Alaska Airlines has 289 airplanes (per wikipedia). All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc... I just can't see an undue strain being placed on a well designed system from Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to…
They also mentioned American was using the same system too. So it may have been load from more than one company.
I don't understand why an airplane being heavier than what was estimated by the software would cause the tail to drag. I would assume it would cause the plane to not have sufficient thrust to take off, or to use more runway. But having the tail drag makes it seem like it had too much thrust, which translated to a steeper take off than anticipated... Can somebody explain?
(Note that you can't keep increasing AoA indefinitely if you need more lift; at a critical speed, the airflow will separate from the wing and stop generating any lift. That's an aerodynamic stall.)
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.
Earlier quoted context omitted.
Could be something as simple as the first concurrency 101 example: atomic addition. The += operator behaves poorly on concurrent access.
We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-oper…
"multiple data streams feeding in weight data per aircraft"? What does that word salad even mean?
Even if what any of what you said was true: these developers are writing critical safety software. If they can't manage to write code without generating race conditions and testing their system for such conditions, they are grossly unqualified to be writing this sort of software.
As others have said: this is a huge fuck-up.
Earlier quoted context omitted.
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the Dyn…
> The purpose of telling pilots thrust was to save money and fuel. No, it's more that using lower thrust during take off saves on engine wear and noise levels around the airport. It doesn't really have that big an impact on fuel use, at least that's not the primary purpose. The article concentrates far too much on the thrust setting. The important bit is the speed where the plane should be rotated to take off, which…
That's no small mistake. It's not a rounding error on the weight of the soda cans loaded into the galley for the flight.
That's most of the passengers. It's most of the freight. It's heavy enough to be a spare engine in the hold. It's a goodsize fraction of the fuel needed to fly from Seattle to Honolulu. It's certainly enough to foul up the mandatory weight-and-balance computation the pilot in command is required to do.
Somehow the input to this software package missed something big. It would be interesting to know exactly what was missed.
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
I was just about to post the _Exact_ same thing. Each execution of the program should be completely independent from other calculations. Something is _Horribly_ wrong with their architecture.
> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide. An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull t…
Yes. Even better if there are situations declared in advance to be automatic halts so no call needs to be made in the moment.