How the Boeing 737 Max disaster looks to a software Developer • Hacker News Board

Re: How the Boeing 737 Max disaster looks to a software Developer

#151

Post by fma » Thu, Apr 18, 2019, 11:17 PM UTC

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.

Re: How the Boeing 737 Max disaster looks to a software Developer

#152

Post by Frost1x » Thu, Apr 18, 2019, 11:20 PM UTC

sisu2019 wrote: post #134

Frost1x wrote: post #129

Earlier quoted context omitted.

From my experience I disagree. It's not that we can't do hard things, it's that there's frequently more profitable approaches than taking difficult challenges. This is becoming systemic across our culture as ideas drawn from capitalism begin invading all aspects of our lives (including how we interact in personal relationships), not just business. Pushing quicker time to market and cost saving solutions to the brink…

Boeing was capitalist from the start. Why fail now in such a stupid way?

Obviously Boeing didn't think they'd have planes falling out of the air (that's bad for business), but I suspect the drive to remain competitive and keep shareholders happy led those further down in the organization to push sloppy solutions (as the article described). That's been my lifelong experience working professionally in the US. It's hard to believe those design oversights mentioned were missed by multiole teams of people.

As the article pointed out, Boeing did their best to keep these software system changes as quiet as they could, so they wouldn't have to go through a new certification process, which is extremely costly (less costly than having planes crash though). That's not something you do when safety is the top priority but it is when the priority is profit.

My central point though is that inherently in the framework of capitalism, the endless pursuit to increase profits by reducing costs (and increasing revenue) will inevitably lead you to the opposite in order to find the optimums you seek. Sometimes, business choices that seem good lead to losses or choices that seem bad lead to gains. You often have to fail (cross the boundaries) in order to see where the boundaries of success lie for various cost saving attributes. All businesses play this game and safety is one of many factors always on the chopping block. Sometimes you have to push safety to see exactly how much investment in safety is truly needed.

Re: How the Boeing 737 Max disaster looks to a software Developer

#153

Post by raz32dust » Thu, Apr 18, 2019, 11:24 PM UTC

Could someone please ELI5 what exactly went wrong that caused the crash? I read the article but I still don't understand if the crash was caused by

(a) The MCAS doing something it wasn't programmed to do (by specification)

(b) MCAS was working as expected, but the expectations/assumptions were wrong (excluding pilot mistakes).

(c) MCAS worked as designed, and the design was correct IF the pilots behaved like Boeing expected them to.

Or has this question not been answered yet? I am talking purely from the software correctness point of view, irrespective of the fact that using software to work around this problem was already bad design.

Re: How the Boeing 737 Max disaster looks to a software Developer

#154

Post by jschwartzi » Thu, Apr 18, 2019, 11:28 PM UTC

AWildC182 wrote: post #117

amluto wrote: post #109

Earlier quoted context omitted.

PID controls do exactly what you are objecting to.

It's more nuanced than that. My ramp functions technically have a += 1 somewhere in them. You just don't want to += a motor value directly or otherwise add to state out of a feedback loop. You can verify the PID function in simulation/unit test. It's much harder to unit test the motor/driver/controller on a stand.

It seems like what you're really getting at is that when you have a control like this where there's a non-linearity in response for a linear change in input quantity, what you really want to do is instead have a non-linear change in input quantity so that there is a linear response. In that case, you would characterize the effect of the control on the response variable and arrive at a table of acceptable values. Then your +=1 becomes an increment of an index which returns the next-highest acceptable value and you no longer produce a non-linear change in the response.

Re: How the Boeing 737 Max disaster looks to a software Developer

#155

Post by philipov » Thu, Apr 18, 2019, 11:28 PM UTC

amelius wrote: post #89

rkagerer wrote: post #49

I believe the relative ease — not to mention the lack of tangible cost — of software updates has created a cultural laziness within the software engineering community. -- This --^ As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. I know folks here will deride me for being "inefficient", but in the long term I still maintain from my experience…

> As someone who carefully crafts their code to strive for perfection, seeing sloppy work out there in the wild drives me nuts. Do you use formal methods to prove the correctness of your code? Because that is what Boeing engineers do (I hope).

"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth

Re: How the Boeing 737 Max disaster looks to a software Developer

#156

Post by ncmncm » Thu, Apr 18, 2019, 11:31 PM UTC

kakwa_ wrote: post #141

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

This analysis is completely right, but in my opinion, focuses too much on the technical aspects. Is MCAS a hack? yes. Is it fixable? yes. Will the 737 MAX continue to fly for two to three decades after all the items above have been addressed? yes. But from an engineering perspective, putting an additional system to "fix" another system feels always a bit weird. Sometimes it's not avoidable (ex: cooling), but when it…

Ultimately it was not a technical failure: the failure was in allowing the plane to be sold with such a thoroughly bad design. Thus, a management and regulatory failure. Regulatory, because FAA signed off on it, obviously without applying any of the process that would have prevented it. Management, because the cost of this debacle will be many, many times what they saved by trying to skate by with a faulty design.

Re: How the Boeing 737 Max disaster looks to a software Developer

#157

Post by m463 » Thu, Apr 18, 2019, 11:34 PM UTC

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

I don't know, I think the 3 points in the article make it glaringly obvious that the root cause is NOT engineering.

The decisions made clearly ignored engineering and historical precedent at every turn.

It's sad because Boeing has had some wonderful engineers, and Boeing aircraft have traditionally allowed the pilots to have the final say.

Re: How the Boeing 737 Max disaster looks to a software Developer

#158

Post by outworlder » Thu, Apr 18, 2019, 11:37 PM UTC

fma wrote: post #151

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.

> Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.

80k for a _light_ in the panel, not to have MCAS take the two sensors into account - which it couldn't do anyway. With two sensors, how do you know which one is right and which one is faulty?

Re: How the Boeing 737 Max disaster looks to a software Developer

#159

Post by ncmncm » Thu, Apr 18, 2019, 11:38 PM UTC

fma wrote: post #151

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.

In court that will cost them dearly: it demonstrates they were aware using two sensors would be safer.

Probably AA or SWA demanded it, and Boeing compromised by charging for it.

But... #6 was about paying attention to the other computer, not the other sensor. That would involve some big and expensive changes to the flight computer software, which they probably should have done long before the MAX project started.

More management failure.

Re: How the Boeing 737 Max disaster looks to a software Developer

#160

Post by 1v1id » Thu, Apr 18, 2019, 11:40 PM UTC

m463 wrote: post #157

ncmncm wrote: post #74

This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops. A second reason is that, under the design still on file at the FAA, it could only add…

I don't know, I think the 3 points in the article make it glaringly obvious that the root cause is NOT engineering. The decisions made clearly ignored engineering and historical precedent at every turn. It's sad because Boeing has had some wonderful engineers, and Boeing aircraft have traditionally allowed the pilots to have the final say.

> So Boeing produced a dynamically unstable airframe, the 737 Max. That is big strike No. 1. Boeing then tried to mask the 737’s dynamic instability with a software system. Big strike No. 2. Finally, the software relied on systems known for their propensity to fail .... Big strike No. 3.

The article definitely does partially blame engineering.