Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

seattletimes.com

Post reply on HN
Print view View original

11–20 of 189 posts

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#11

Post by Tempest1981 »

Curious about the actual bug:

> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system

> the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings.

Multithreading/contention issue? But how would that alter the weights?

> The software code was permanently repaired about five hours later

That's surprisingly fast, isn't it?

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#12

Post by Aloha »

sidlls wrote: post #7
Aloha wrote: post #2

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think eve…

This is a failure in the development of the software that good process in the purchaser’s handbook saved them from. That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.

They may have thought their test coverage, covered it adequately - who knows, the success was on Alaska's part.

Also process, procedure and more importantly people are always the last line of defense against failure.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#15

Post by cratermoon »

sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#16

Post by sidlls »

Tempest1981 wrote: post #11

Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system > the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings. Multithr…

Multithreading/contention issue? But how would that alter the weights?

Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#17

Post by fwlr »

“the bug was missed because it only presented when many aircraft at the same time were using the system.”

The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load?

The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#18

Post by basch »

sudhirj wrote: post #10

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#19

Post by guitarbill »

hodgesrm wrote: post #5

That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.

Agreed, although it takes an individual with a huge amount of integrity or a culture that values and promotes integrity to make a call like this:

> “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.”

Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing was wrong.

Post reply on HN