Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How the Boeing 737 Max disaster looks to a software Developer

spectrum.ieee.org

Post reply on HN
Print view View original

161–170 of 306 posts

Re: How the Boeing 737 Max disaster looks to a software Developer

#161

Post by gizmo385 »

d-sc wrote: post #112
ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

Architecting solutions is hard. In this case, you need knowledge of motors, flight controls, sensor fusion, etc. It’s easy to find edge cases when they present themself (tragically here). But most electrical-mechanical-software assemblies have similar issues.

> In this case, you need knowledge of motors, flight controls, sensor fusion, etc.

Expertise which should be readily available at one of the world's foremost designers of high performance commercial aircraft.

Re: How the Boeing 737 Max disaster looks to a software Developer

#162

Post by ncmncm »

m463 wrote: post #157
ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

I don't know, I think the 3 points in the article make it glaringly obvious that the root cause is NOT engineering. The decisions made clearly ignored engineering and historical precedent at every turn. It's sad because Boeing has had some wonderful engineers, and Boeing aircraft have traditionally allowed the pilots to have the final say.

Of course, that much engineering failure is not possible without even more management and regulatory failure. But documenting the engineering failures is necessary to quantify the management failures, at least until discovery in the wrongful-death lawsuits begins.

Re: How the Boeing 737 Max disaster looks to a software Developer

#163

Post by andbberger »

alacombe wrote: post #76
andbberger wrote: post #67

Earlier quoted context omitted.

[citation needed] idk man, my code is pretty damn clean

Code cleanliness has no correlation with lack of bugs. Also, 1) your sample size is pretty small, and 2) what looks clean to you might very well be despicable to someone else.

I guess that was a poorly chosen word on my part then. How about 'elegant'.

Re: How the Boeing 737 Max disaster looks to a software Developer

#164

Post by bumby »

blunte wrote: post #107

I think a fundamental concept that is relatively ignored in press (mainstream or non-fringe at least) is in business priorities. As the essay points out, Boeing made decisions based on market (financial) factors all along the way. Of course they did - almost all companies do. Because it's so common, we forget it's not a RULE that cost, shareholder value, profit, etc. must be the final judgement. Certainly any company…

I agree that people often seem to miss that this may be rooted in a business decision. The problem is it isn't a risk-informed decision. I would doubt Boeing was accurately able to assess the actual risk of MCAS causing a catastrophic failure or else the decision to rush to market wouldn't have happened.

I think just as large of a problem is mis-aligned incentives. Management is almost assuredly not singly focus on the extreme long term. The are graded quarter-by-quarter, year-by-year. This pushes uncertain risks to the periphery in favor of short term profits. I'm worried that unless the incentive structure is re-evaluated (by jail terms as an example) management will continue to make these types of decisions because schedule and cost will remain king

Re: How the Boeing 737 Max disaster looks to a software Developer

#165

Post by kortilla »

outworlder wrote: post #158
fma wrote: post #151

Earlier quoted context omitted.

Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.

> Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor. 80k for a _light_ in the panel, not to have MCAS take the two sensors into account - which it couldn't do anyway. With two sensors, how do you know which one is right and which one is faulty?

Well you can’t get quorum but you can certainly shut off when you detect a conflict.

Re: How the Boeing 737 Max disaster looks to a software Developer

#166

Post by outworlder »

> When MCAS senses that the angle of attack is too high, it commands the aircraft’s trim system (the system that makes the plane go up or down) to lower the nose. It also does something else: It pushes the pilot’s control columns (the things the pilots pull or push on to raise or lower the aircraft’s nose) downward.

Wait a minute. Since when does MCAS include a stick pusher? I haven't flown anything outside simulators yet, but AFAIK 737's do not have stick pushers of any kind.

Re: How the Boeing 737 Max disaster looks to a software Developer

#167

Post by ncmncm »

acqq wrote: post #139
ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

> A sixth was that, besides comparing redundant sensors, it could have compared what the other flight computer thought it should be doing. If you mean AoA sensors, AFAIK, there was absolutely no redundancy at all the way MCAS was designed. Exactly one sensor was ever used for MCAS. And last time I've read the Boeing's reported coming software changes, they wanted to keep it so, but just to add the notification to the…

Comparing sensors and comparing judgment of the other computer were both things they could have done, that they utterly failed to do, in both cases.

Re: How the Boeing 737 Max disaster looks to a software Developer

#168

Post by quickthrower2 »

adreamingsoul wrote: post #75
rkagerer wrote: post #49

I believe the relative ease — not to mention the lack of tangible cost — of software updates has created a cultural laziness within the software engineering community. -- This --^ As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. I know folks here will deride me for being "inefficient", but in the long term I still maintain from my experience…

Recently, I had someone join my team who was instantly popular and everyone enjoyed this person's personality. I trusted this person more then I should have and during a period when I was unable to give enough attention to their code reviews, they introduced several critical bugs and defects into the production environment. I never figured out if it was ignorance or laziness. But, either way I learned a valuable less…

They are new to the team and are not getting sufficient code reviews. I wouldn’t put the blame on them. There is a lot of gotchas in an unfamiliar codebase. Although they probably should have known this and insisted on a full review if they are at senior level.

Re: How the Boeing 737 Max disaster looks to a software Developer

#169

Post by ncmncm »

DuskStar wrote: post #146

This article reads well. Unfortunately, it's filled with fundamental mistakes. > In the old days, when cables connected the pilot’s controls to the flying surfaces, you had to pull up, hard, if the airplane was trimmed to descend. You had to push, hard, if the airplane was trimmed to ascend. With computer oversight there is a loss of natural sense in the controls. In the 737 Max, there is no real “natural feel.” Whoo…

Yes, it is unfortunate that these errors detracted from the thrust of the article.

Re: How the Boeing 737 Max disaster looks to a software Developer

#170

Post by kortilla »

jschwartzi wrote: post #154
AWildC182 wrote: post #117

Earlier quoted context omitted.

It's more nuanced than that. My ramp functions technically have a += 1 somewhere in them. You just don't want to += a motor value directly or otherwise add to state out of a feedback loop. You can verify the PID function in simulation/unit test. It's much harder to unit test the motor/driver/controller on a stand.

It seems like what you're really getting at is that when you have a control like this where there's a non-linearity in response for a linear change in input quantity, what you really want to do is instead have a non-linear change in input quantity so that there is a linear response. In that case, you would characterize the effect of the control on the response variable and arrive at a table of acceptable values. Then…

No, you’re missing the point a bit. It’s not about the magnitude of the response, it’s about the propensity for a feedback loop.

=+1 is shorthand for “ignore everything going on in the world and increase your value”. This is almost never what anyone actually wants so they try to spend a bunch of time guarding against calling that when it’s already at a maximum.

Instead, the safe thing to do is only assign to it from a function with a ceiling.

val = min(CEIL, val+1)

It’s way to easy to get runaways with =+1 even in serious systems like this one. Every time I see that in code I review where the value is some long-lived thing, I just confirm with the author that they don’t care if it overflows, because it’s probably gonna happen.

Post reply on HN