Live data from Hacker News

Boeing 737 MAX crash and the rejection of ridiculous data

philip.greenspun.com

181–190 of 194 posts

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#181

It should be noted in the doomed Air France 447 flight, the plane activated the stall warning because of a high angle of attack that was leading to stall. (thanks pdx for the corrected info) At some point the system rejected the data and stopped the stall warning because the angle of attack was so severe that it considered the data erroneous. This is speculated to have caused the co-pilot to keep pulling back on the…

The other problem with AF447 was that the pilot and copilot were giving conflicting instructions to the plane and there wasn't any feedback that they were doing so, and that the senior pilot was taking a nap...

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#182

This video from "Mentour pilot" has been deleted (supposedly upon request from boeing): https://www.youtube.com/watch?v=EzgBft-79U8 You can see it here (european version of youtube): https://vimeo.com/329558134 Or more info here: https://news.ycombinator.com/item?id=19627525

Strongly suggest everyone check out the video. It's...kind of shocking.

It's also misleading. The pilot pulling back on the stick makes the wheel harder to turn. If he stopped doing that and helped turn the trim wheel they would have been able to adjust it.

All that video really shows is that if you do it wrong it doesn't work. Not all that shocking

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#183
post #150

Earlier quoted context omitted.

It is difficult to be sure from the flight data chart that was published in the Seattle Times, but it looks possible that the faulty AofA reading was below 25 degrees for most of the Lion Air flight. Reading between the lines, Greenspun has presented these crashes as "the first mass killings by software". That claim looks clearer if you present it as wholly due to the absence of a few common-sense lines of code, and…

From my pilot perspective, if a fellow pilot took the action MCAS did, I would consider it homicidal. This is based on the last instance of MCAS activation at ~05:43:21, airspeed is already exceeded maximum operating speed (Vmo), MCAS commands 1.3 units of nose down trim inside of 5 seconds, which changed the attitude to ~18 degrees nose down within 6 seconds. The vertical acceleration momentarily went negative, mean…

To be clear, I do not object to the 'verdict', but I do not think it helps to suggest that all the issues raised by these crashes can be dealt with just by giving each sensor one or two somewhat arbitrary thresholds beyond which they should be ignored. While I cannot think of any reason not to have adopted the proposed solution within MCAS, and it would have prevented the Ethiopian Airlines crash, it would not have eliminated the possibility in other circumstances (possibly including Lion Air), on account of the other problems with MCAS. If, in these cases, the sensors had failed to a plausible value (say, 15 degrees), the outcome would have been the same.

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#184
post #119

All the talk about how crappy Boeing engineering here was is bullshit and speculation and I am surprised PG participates in it. What we can discuss objectively here is incident response in which Boeing allowed the situation to continue after the first crash. How did they not run hundreds of hours of simulations, code reviews etc, etc on the system assumed to be at fault? How did they not immediately change the safety…

It's not really speculation. The proof, as they say, is in the impact crater. The only mystery left is, what is the nature of the paper trail that led to this catastrophe? Was there malicious malfeasance? Overt and irresistible pressure to certify at all costs? Was it all just a tragic mistake? We don't know. We only know the physical systems that contributed to the crashes, and some of the motivations that would hav…

"The technical implementation can be roughly inferred by any programmer", "such a foreseeable failure state"... how many years of experience do you have?

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#186
post #184

Earlier quoted context omitted.

It's not really speculation. The proof, as they say, is in the impact crater. The only mystery left is, what is the nature of the paper trail that led to this catastrophe? Was there malicious malfeasance? Overt and irresistible pressure to certify at all costs? Was it all just a tragic mistake? We don't know. We only know the physical systems that contributed to the crashes, and some of the motivations that would hav…

"The technical implementation can be roughly inferred by any programmer", "such a foreseeable failure state"... how many years of experience do you have?

How rude. Here I was thinking we were having a civil discourse over the Internet. More than born yesterday, less than since the Moon landing.

Regardless, my assessment is based on most juniors I've worked with. By their third year most seem to have already grasped the need to test for boundary conditions, and to ensure proper error handling for GIGO failure. Any 1 year+ with at least FizzBuzz levels of understanding can be handheld toward it with the right nudge, and in fact, the less experience they have the more eager and likely they tend to be to pick up on error handling since they haven't yet developed sufficient skills to be able to get their head around the "test you don't need to write" because the result can be inferred from a test at another level of the system (a frequent coping strategy that starts creeping it's way in with increased levels of familiarity with a complex system).

Any problem grokking the above points is usually solved with an impromptu exercise and lecture where I have the junior play the part of a computer until they realize just how much the computer "doesn't know", and has no capacity to derive from reasoning, unless it's actually coded/implemented to. I've not yet had a junior who failed to grasp this to some degree (though a recent one is giving me a run for my money), and become capable within a couple months of inferring two to three-function away error states to test for. Within the year, I can typically point them at an arbitrary code block and get back a reasonable testing surface.

Which brings me to my next observation, where I think you may be attempting to make a point:

If I run into juniors of 1-3 years experience who need coaching to fully understand what I explained above, then perhaps the average programmer is not capable of inferring what I claim.

To which all I can say is, my observation may be skewed, because I'm a bloody paranoid polyglot of a tester when it comes to safety critical systems. Even when I was pre-collegiate programming calculators, the more someone else actually depended on something, the greater the lengths I'd go through to test things before cutting them loose with anything I was producing for them. The THERAC-25 postmortem is bedtime reading for me, and I've pushed myself to understand computer science and software engineering as more than mere 'coding'.

If the argument then, is that I'm an atypical representative of my software composing brethren, then I'd like to know why in the $deity's name we're not triple-checking safety critical code at system integration time, seeing as we can assume this level of inattention to detail by the average programmer. Especially given as the languages these types of systems are implemented in are typically not the most 'friendly' languages.

This suggests cultural issues, undue pressure to fast-track approval, disincentive to raise red flags that could impede delivery, or an "over-the-wall" hyper siloing of expertise/responsibility that lead to the least experienced in complex system implementation being blindly trusted by those who had the experience to realize something was horribly wrong.

If the above doesn't assuage any concerns relating to my experience, I'm afraid not much else will.

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#187
post #150

Earlier quoted context omitted.

From my pilot perspective, if a fellow pilot took the action MCAS did, I would consider it homicidal. This is based on the last instance of MCAS activation at ~05:43:21, airspeed is already exceeded maximum operating speed (Vmo), MCAS commands 1.3 units of nose down trim inside of 5 seconds, which changed the attitude to ~18 degrees nose down within 6 seconds. The vertical acceleration momentarily went negative, mean…

To be clear, I do not object to the 'verdict', but I do not think it helps to suggest that all the issues raised by these crashes can be dealt with just by giving each sensor one or two somewhat arbitrary thresholds beyond which they should be ignored. While I cannot think of any reason not to have adopted the proposed solution within MCAS, and it would have prevented the Ethiopian Airlines crash, it would not have e…

[deleted]

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#188
post #150

Earlier quoted context omitted.

It is difficult to be sure from the flight data chart that was published in the Seattle Times, but it looks possible that the faulty AofA reading was below 25 degrees for most of the Lion Air flight. Reading between the lines, Greenspun has presented these crashes as "the first mass killings by software". That claim looks clearer if you present it as wholly due to the absence of a few common-sense lines of code, and…

From my pilot perspective, if a fellow pilot took the action MCAS did, I would consider it homicidal. This is based on the last instance of MCAS activation at ~05:43:21, airspeed is already exceeded maximum operating speed (Vmo), MCAS commands 1.3 units of nose down trim inside of 5 seconds, which changed the attitude to ~18 degrees nose down within 6 seconds. The vertical acceleration momentarily went negative, mean…

If you view the MCAS as a control system that is a safety component of the 737 MAX, then you need a really good reason to completely cut off that safety system when a particular sensor input goes above a given threshold. In this case the argument being made is that if the input is above 24 degrees the safety system should be turned off, potentially leading to a stall. It's almost an argument against having that system entirely.

The problem is much more complicated than this and requires thinking end to end about 1) what the purpose of the system is, 2) when and how it should operating, 3) how much control it should have, 4) how it's activity is made visible to the pilot when it performs any control, 5) how and under what conditions it should automatically disable itself, or 6) be able to be manually disabled, and 7) how the pilot is made aware of all of these situations in a way that doesn't cause confusion in potentially complex scenarios involving other failures and alarms going off, and 8) proper training so the pilot can manage the plane when the characteristics have changed after the system is disabled. Oh, and 9) in this case since it's impossible for a pilot to manually trim a 737 when it's above a couple of hundred knots, making sure an electronic trim assistance function is available in this scenario (which it wasn't due to tragic idea to overload the use of the trim runaway cutout switches).

And I am probably missing a bunch of things, which is part of the point.

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#189
post #184

Earlier quoted context omitted.

"The technical implementation can be roughly inferred by any programmer", "such a foreseeable failure state"... how many years of experience do you have?

How rude. Here I was thinking we were having a civil discourse over the Internet. More than born yesterday, less than since the Moon landing. Regardless, my assessment is based on most juniors I've worked with. By their third year most seem to have already grasped the need to test for boundary conditions, and to ensure proper error handling for GIGO failure. Any 1 year+ with at least FizzBuzz levels of understanding…

You think I was rude asking you about your experience. Now think how rude this unsubstantiated allegation of obvious simplicity of the code in question is to the person who wrote it -- with the weight of hundreds of lost lives on their shoulders. These control systems can get arbitrary complex. We don't know anything about the hardware this runs on and what it has to interface with. We don't know the constraints and age of the codebase. Nothing. To assume that this boils down to a simple if statement is something I would expect from a recent college graduate, or someone who has only worked at a web startup, not a person with 5+ years of real world experience building complex systems. I agree about all the points about testing and business processes. We have enough evidence to conclude that unforgivable mistakes were made there (and I point to that in my original comment).

Re: Boeing 737 MAX crash and the rejection of ridiculous data

#190
post #34

Earlier quoted context omitted.

Unless I'm wrong disabling the MCAS system leaves the elevator at the current trim as set by the jack screw. That's really not the behavior you want. You want the control to return to neutral. Which probably isn't the way the trim mechanism is designed. This whole thing smells like a bunch of dodgy decisions. Some of them recent and others historical. For instance not being able to manually adjust trim while the elev…

There's no way to "disable MCAS". There's just "disable the stabilizer trim motor". At which point the motor can't help you restore safe trim.

Yes that's more or less it from what I gather. The trim cutout is designed to stop runaway from due to failure of the motor control circuit. Some ommon failure modes leaves the output stuck on. The only fix is to kill the power.

However if you kill the power to the trim motor and then turn it back on MCAS will blindly add more trim on top of the trim it previously added. It doesn't remember how much trim it's already applied.

Post reply on HN