Live data from Hacker News

How the Boeing 737 Max disaster looks to a software Developer

spectrum.ieee.org

61–70 of 306 posts

Re: How the Boeing 737 Max disaster looks to a software Developer

#61
Couple pedantic quibbles, forgive me:

>I will leave a discussion of the corporatization of the aviation lexicon for another article, but let’s just say another term might be the “Cheap way to prevent a stall when the pilots punch it,” or CWTPASWTPPI, system. Hmm. Perhaps MCAS is better, after all.

It's not actually just intended for when pilot's pour on the thrust, it's for any time they pour on the AoA. There need not be any throttle change involved to bring this about. The most readily imagined example, however, is low speed, large throttle increase like you'd have if you were aborting a landing attempt to go around.

>When MCAS senses that the angle of attack is too high, it commands the aircraft’s trim system (the system that makes the plane go up or down) to lower the nose. It also does something else: It pushes the pilot’s control columns (the things the pilots pull or push on to raise or lower the aircraft’s nose) downward.

Nothing I've read characterizes MCAS as having an inbuilt-stick pusher. I've mentioned before that MCAS shares a spot with stick-pushers in terms of what they are trying to do, and problem they are employed to solve but MCAS does not actively put force on the stick through an extra mechanism with the intent to actuate a control surface deflection, and alerting the pilot through the haptic response, which is the defining characteristic of that type of system as I understand it. All MCAS does is modify the trim, which has the effect of passively modifying the flight characteristics of the aircraft.

>In the 737 Max, like most modern airliners and most modern cars, everything is monitored by computer, if not directly controlled by computer. In many cases, there are no actual mechanical connections (cables, push tubes, hydraulic lines) between the pilot’s controls and the things on the wings, rudder, and so forth that actually make the plane move. And, even where there are mechanical connections, it’s up to the computer to determine if the pilots are engaged in good decision making (that’s the bitey dog again).

As far as I am aware, Boeing has maintained manual reversion with regards to the trim system and yoke in the 737 MAX 8. I.e. there are direct connections to the control surfaces from the pilot's hand actuated controls. Boeing's concession to FBW is by implementing parallel automation managed control circuits that allow for electrically driven manipulation of control surfaces fed back to the pilot through the mechanical linkage. The envelope-protection -> bitey-dog analogy in this case, is still an accurate characterization.

>But it’s also important that the pilots get physical feedback about what is going on. In the old days, when cables connected the pilot’s controls to the flying surfaces, you had to pull up, hard, if the airplane was trimmed to descend. You had to push, hard, if the airplane was trimmed to ascend. With computer oversight there is a loss of natural sense in the controls. In the 737 Max, there is no real “natural feel.”

As far as I can ascertain, there still is "natural feel" in regards to pitch control. There are no hydraulic boosters in place. The problem though, is that MCAS actually hides the aberrant behavior at high AoA from the pilot by intentionally down-trimming the plane. This gets the job done (by some definitions), but does it instead through the trim system (which operates automatically in other circumstances enough where the pilot may mistake the behavior for some other system) and without doing anything to the stick (the primary instinctual control for the plane).

Other than those quibbles, this is a beautiful write-up that is a treat to read. The Supplemental Type Certification section was also informative for me, as it illustrates unambiguously that there was a process to handle this exact type of augmentation which was sidestepped by Boeing for whatever reason.

Bravo!

Re: How the Boeing 737 Max disaster looks to a software Developer

#62
The 737 Max keeps getting viewed as an "Engineering Failure" but we should consider if this really was a "Management" failure. The unstated goal w/MCAS was to avoid additional pilot type certification. (If you tell pilots about MCAS, give them an off switch, or if the system switches off... then pilots need to be trained specifically on the MAX.) If management gave the MCAS project to Senior Boeing Engineers they would likely push back jeopardizing unstated goals. Executives likely steered this project to junior engineering or yes-men... who delivered a solution which avoided pilot training.

Re: How the Boeing 737 Max disaster looks to a software Developer

#63
"Neither such coders nor their managers are as in touch with the particular culture and mores of the aviation world as much as the people who are down on the factory floor, riveting wings on, designing control yokes, and fitting landing gears. Those people have decades of institutional memory about what has worked in the past and what has not worked. Software people do not."

I wonder if, once basic programming becomes a part of standard education, these kinds of problems will be mitigated. Right now if you're a developer, you're not anything else. So developers come in as outsiders to any given industry and have to learn about it. But software is applicable to nearly every industry, so we end up with lots of domain-specific code written by people who don't know those domains very well. What if the domain experts could not only speak the language of code, but could do some of the coding themselves?

Re: How the Boeing 737 Max disaster looks to a software Developer

#64

I believe the relative ease — not to mention the lack of tangible cost — of software updates has created a cultural laziness within the software engineering community. -- This --^ As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. I know folks here will deride me for being "inefficient", but in the long term I still maintain from my experience…

The most efficient way to do it is to do it once.

Re: How the Boeing 737 Max disaster looks to a software Developer

#65

I believe the relative ease — not to mention the lack of tangible cost — of software updates has created a cultural laziness within the software engineering community. -- This --^ As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. I know folks here will deride me for being "inefficient", but in the long term I still maintain from my experience…

If only managers and CEOs saw it the same way.

You'll see it in any piece of software written by any developer who is self-managed. The notion that software quality would greatly improve without managers is a myth.

Re: How the Boeing 737 Max disaster looks to a software Developer

#66

I believe the relative ease — not to mention the lack of tangible cost — of software updates has created a cultural laziness within the software engineering community. -- This --^ As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. I know folks here will deride me for being "inefficient", but in the long term I still maintain from my experience…

If only managers and CEOs saw it the same way.

One the other side, unless you are an artist and not dependant on your software actually being used, you have to get product out at some point.

It's easy to demonize CEO and managers. We, as devs, will always find an excuse as to why the software is not ready and QA will always find new things to test to justify not putting the software out.

Re: How the Boeing 737 Max disaster looks to a software Developer

#67

Earlier quoted context omitted.

If only managers and CEOs saw it the same way.

You'll see it in any piece of software written by any developer who is self-managed. The notion that software quality would greatly improve without managers is a myth.

[citation needed]

idk man, my code is pretty damn clean

Re: How the Boeing 737 Max disaster looks to a software Developer

#68
So, the linked article is an updated version of an earlier article in the EE times. In the comments section, a reader (ie) Frankly, I am astonished that a single point of failure (AOA) could make it through a FMEA (Failure Mode Effect Analysis). For those who are unfamiliar, a FMEA is (basically) looking at each part of a system and saying "What happens if it breaks?". Having worked in commercial vehicle software development, I am astonished as well.

Re: How the Boeing 737 Max disaster looks to a software Developer

#69
post #37

Earlier quoted context omitted.

I think this is likely to be true, but I do wonder if there might be a better way to spend those resources. For this particular example, you’d have been better off putting more money into flight testing with various sensor errors.

Well, yes. But, and who knows if it is true, but you would think you can think up that you need sensor errors to be tested and maybe figure out you need three errors vs two for instance to cover all cases; formal systems can help you model and reason about that and come up with cases that are not covered. I think the point is which these disasters show, and some others in the past (was there not a recalled Japanese c…

Two sensors is enough if you can shut off the system when they disagree. MCAS didn’t doom the plane if it shut off, so that was enough. The problem was that they didn’t do this, and instead used only one sensor at a time. Basic sense and standard practice tells you not to do that. I don’t know if more formality would have helped them not do something so boneheaded.

However, this looks like an unusual case. Based on the fact that they had such a dumb design for this system, that shouldn’t have passed muster in any aviation context no matter your software methodology, it’s probably not representative enough to draw a larger lesson about engineering. The larger lesson seems to be business and regulatory.

Regarding Japanese cars, Toyota went through a big thing with unintended acceleration that got quite a few people killed. Software was suspected, but the cause was ultimately determined to (mostly?) be a combination of drivers mixing up the accelerator and the brake (happens more often than you might think) and unsecured floor mats pushing the accelerator down. I bought a Toyota not too long after this and the dealer made a special point to show me how the floor mat attached to the floor and that I must be sure it was solidly connected.

Their software was audited and apparently it was really badly made. It was described as “spaghetti-like” and had global variable abuse, ignored errors, failed to restart unresponsive tasks, and had potential memory corruption problems. It’s quite possible that this really was the cause, and a jury even found this to be the cause in a civil trial over one of the crashes, but it was never definitely linked.

Re: How the Boeing 737 Max disaster looks to a software Developer

#70
post #20

Earlier quoted context omitted.

Interesting quote. Seems it was actually a quote of Henry Spencer by Carmack. http://number-none.com/blow/john_carmack_on_inlined_code.htm... Note however that there were several software-related crashes in the early days of Gripen. Might not be related to exactly this code, but to the problem of regulating the feedback loops to manage the plane's purposeful instability

huh? interesting, do you have more info, I love reading stuff like this. I don't know much about the Gripen but it is my all time favorite after the F-16. Off topic but here's a neat youtube video of Gripen jets undergoing rapid turn-around, on a normal street road with just a handful of people, and special tools for the crew: https://www.youtube.com/watch?v=49L9BlYQSjw

The first crash: https://www.youtube.com/watch?v=k6yVU_yYtEc

Second crash (over middle of Stockholm) https://youtu.be/mkgShfxTzmo?t=133

From Wikipedia: During the test programme, concern surfaced about the aircraft's avionics, specifically the fly-by-wire flight control system (FCS), and the relaxed stability design. On 2 February 1989, this issue led to the crash of the prototype during an attempted landing at Linköping; the test pilot Lars Rådeström walked away with a broken elbow. The cause of the crash was identified as pilot-induced oscillation, caused by problems with the FCS's pitch-control routine.[22][33][34]

In response to the crash Saab and US firm Calspan introduced software modifications to the aircraft. A modified Lockheed NT-33A was used to test these improvements, which allowed flight testing to resume 15 months after the accident. On 8 August 1993, production aircraft 39102 was destroyed in an accident during an aerial display in Stockholm. Test pilot Rådeström lost control of the aircraft during a roll at low altitude when the aircraft stalled, forcing him to eject. Saab later found the problem was high amplification of the pilot's quick and significant stick command inputs. The ensuing investigation and flaw correction delayed test flying by several months, resuming in December 1993.[22]

Post reply on HN