After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’ • Hacker News Board

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#141

Post by justinclift » Sat, Feb 18, 2023, 7:59 PM UTC

speed_spread wrote: post #80

cratermoon wrote: post #15

Earlier quoted context omitted.

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.

Boeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg:

https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled)

https://archive.is/vdg9S (paywall workaround)

Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#142

Post by t0mas88 » Sat, Feb 18, 2023, 8:08 PM UTC

Aloha wrote: post #2

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think eve…

> Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable

Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are free to deviate from reduced thrust calculations and will do so.

This incident is much more about the calculated speeds to use and aircraft trim settings. That's what caused the tail strikes, trying to lift off at a speed that is too low for the weight.

Those numbers are required by every airliner taking off. And getting them from software is standard everywhere. The alternative is doing the calculations manually using paper charts, every pilot is required to know how to do that, but nobody does it that way for commercial operations.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#143

Post by ceejayoz » Sat, Feb 18, 2023, 8:12 PM UTC

bogomipz wrote: post #105

The article states the following about the software: >"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity. >In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be read…

"Regularly maxed-out" translates to "fairly predictable". If the 20 people in first class all have 500 pounds of uranium each they've snuck onboard, you have a problem, but otherwise, there's plenty of wiggle room to account for minor variability of this kind.

They'll have a decent idea of the average weight of the average passenger (https://airinsight.com/the-pending-new-faa-weight-balance-ru...). If you're chartered by an anorexia treatment facility to move 300 patients, you may put a little extra attention to the exact values.

> The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This includes an extra ten pounds for winter and five pounds for summer. This also includes 16 pounds for personal items, up from ten. Airlines must increase the average weight of female passengers and their carry-ons from 145 pounds to 179 pounds in summer, and from 150 pounds to 184 pounds in winter. The average weight for males with carry-ons is increased from 185 pounds in summer to 200 pounds, and from 190 pounds to 205 pounds in winter.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#144

Post by raldi » Sat, Feb 18, 2023, 9:05 PM UTC

hef19898 wrote: post #130

raldi wrote: post #92

Earlier quoted context omitted.

Yes. Even better if there are situations declared in advance to be automatic halts so no call needs to be made in the moment.

No matter what complex system it is, you always need humans in the loop. Especially for networks and not just single nodes.

I think you misunderstand my comment. I'm saying, for instance, that during a calm period when a group of people can take their time and review each other's reasoning, a rule might be put in effect like, "Two tail strikes within X hours of each other means an automatic, immediate halt." Humans execute that procedure (or override it at the risk of their career) and then decide what to do next, having been in the loop the entire time.

Such a rule likely saved an SR-71 once: https://theaviationgeekclub.com/the-story-of-belmont-86-the-...

The natural tendency for military air crews is to complete the mission if humanly possible. To counter this inclination, the Wing Commander had designated certain emergencies sufficiently critical to require immediate landing. This was one of those emergencies.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#145

Post by AdrianB1 » Sat, Feb 18, 2023, 9:28 PM UTC

phkahler wrote: post #127

mnw21cam wrote: post #116

Earlier quoted context omitted.

> The purpose of telling pilots thrust was to save money and fuel. No, it's more that using lower thrust during take off saves on engine wear and noise levels around the airport. It doesn't really have that big an impact on fuel use, at least that's not the primary purpose. The article concentrates far too much on the thrust setting. The important bit is the speed where the plane should be rotated to take off, which…

It seems like a slower rotation might be useful, to wait until you feel it leave the ground before aiming higher. I'm sure there are reasons for the procedure as is.

Rotating at a slower speed is a bad idea. Indicating a lower weight, resulting in a lower calculated rotating speed, is the root cause of the problem. There is no "slower rotation", one does not just pull the yoke slowly, raise the nose wheel and wait for the plane to take off by itself when it will be fast enough, that you can do on a light prop plane where rotate speed is so low some planes will take off by themselves, but not on an airliner where you have a high chance that raising the nose too early results in speed not increasing or even decreasing.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#146

Post by Aloha » Sat, Feb 18, 2023, 9:33 PM UTC

t0mas88 wrote: post #142

Aloha wrote: post #2

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think eve…

> Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are fr…

Thank you for the thoughtful comment!

I thought I was missing something in the way the story read, I'm glad to know I was.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#147

Post by AdrianB1 » Sat, Feb 18, 2023, 9:34 PM UTC

Ozzie_osman wrote: post #42

hn_throwaway_99 wrote: post #26

Earlier quoted context omitted.

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to…

True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof". If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).

In most industries that are not IT, testing the IT systems very well is needed, but rarely achieved. In my area (FDA regulated) performance testing is almost never done correctly and (for sure) never completely, but as long as it works and the auditors never look at these details nobody from management is willing to know about it, because fixing it costs money. For many years the people signing off for all the software were intentionally not business people with zero IT knowledge, so they could not be accused of knowingly signing off low quality code. That saved tens or hundreds of millions of dollars in the past 20 years alone.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#148

Post by dandandan » Sat, Feb 18, 2023, 9:35 PM UTC

AdrianB1 wrote: post #145

phkahler wrote: post #127

Earlier quoted context omitted.

It seems like a slower rotation might be useful, to wait until you feel it leave the ground before aiming higher. I'm sure there are reasons for the procedure as is.

Rotating at a slower speed is a bad idea. Indicating a lower weight, resulting in a lower calculated rotating speed, is the root cause of the problem. There is no "slower rotation", one does not just pull the yoke slowly, raise the nose wheel and wait for the plane to take off by itself when it will be fast enough, that you can do on a light prop plane where rotate speed is so low some planes will take off by themsel…

I interpreted "slower" as "later," ie wait longer before rotating.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#149

Post by AdrianB1 » Sat, Feb 18, 2023, 9:41 PM UTC

speed_spread wrote: post #80

cratermoon wrote: post #15

Earlier quoted context omitted.

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.

I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.

Re: After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

#150

Post by AdrianB1 » Sat, Feb 18, 2023, 9:42 PM UTC

klyrs wrote: post #91

howinteresting wrote: post #77

Earlier quoted context omitted.

Should have used Rust.

A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.

Bad coding happen even in transactional databases, see my example in the other comment.