Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

291–300 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#291

Post by vikingerik »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

When that risk-adverse culture is carried over to another space, it just fails and never gets noticed. Risk-adverse organizations get outcompeted by more efficient risk-tolerant ones, every time that such competition is possible.

Risk aversion only works for an entity that has a forcible monopoly in its space. The FAA and FDA do. Another is the Nuclear Regulatory Commission, which exhibits the same behavior: their job is to prevent accidents, and the surest way to do that is to never approve anything at all.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#292

Post by dmix »

rmrfchik wrote: post #121
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

Just my 2 cents: 2 billons are payed not for reliability but for size. Size of the system (and size of the team).

$2 billion is an insane amount of money even for a major gov software project

And even after $2B it was completely broken and late and required another year and hundreds of millions.

I get the feeling people see thousands of millions of dollars as some abstract thing. You could build an A+ team with 1/100th that cash. Yet it still ends up sucked down a blackhole that's only designed to consume more money.

And there's zero consequences for failure. The same few contractors will get the contract next time.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#293

Post by spearman »

brazzy wrote: post #278
gnarbarian wrote: post #257

Earlier quoted context omitted.

a few things 1) mostly thank the airline operators for that. 2) the fact the 737 Max disasters happened outside the US was a matter of chance not exemplary policies by the FAA. FAA policies did not prevent the deployment and roll out of this dangerous aircraft in the United States and did not lead to grounding of the aircraft until multiple events occurred. 3) I'm mostly talking about software here, and I believe a b…

> 1) mostly thank the airline operators for that. This is a 100% proof that you have not the faintest clue what you're talking about. The #1 priotity of airline operators is profit, not safety.

Another way to think about it is that safety is good for profits, at least in so far as consumers care about it. And I think consumers do care quite a bit. Of course there are arguments to be made for regulation, but I think your statement is overly emphatic.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#294

Post by _ea1k »

rattlesnakedave wrote: post #244
Someone1234 wrote: post #214

Earlier quoted context omitted.

"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. Obviously the NOT…

> This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. This stuff is such cringe, it is the type of response you see from jaded boomers more focused on box-ticking and punching out as opposed to doing anythin…

The problem with replacing NOTAMs is that it actually is a fairly simple system that works pretty well from a technical standpoint.

The biggest issues in general relate to the data itself and the processes involved.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#295

Post by neilv »

JCM9 wrote: post #231
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

This demonstrates a gross misunderstanding of how the FAA actually operates in its efforts to address “risk.” It’s far more theatre via bureaucracy and paperwork than actual proven engineering efforts that demonstrably reduce risk.

On what is your own understanding based?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#296

Post by justinsaccount »

alexb_ wrote: post #271
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#297

Post by gnarbarian »

brazzy wrote: post #278
gnarbarian wrote: post #257

Earlier quoted context omitted.

a few things 1) mostly thank the airline operators for that. 2) the fact the 737 Max disasters happened outside the US was a matter of chance not exemplary policies by the FAA. FAA policies did not prevent the deployment and roll out of this dangerous aircraft in the United States and did not lead to grounding of the aircraft until multiple events occurred. 3) I'm mostly talking about software here, and I believe a b…

> 1) mostly thank the airline operators for that. This is a 100% proof that you have not the faintest clue what you're talking about. The #1 priotity of airline operators is profit, not safety.

I was a software engineering manager at the FAA for more than six years.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#298

Post by bastardoperator »

haswell wrote: post #290
bastardoperator wrote: post #282

Earlier quoted context omitted.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…

We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#299

Post by _ea1k »

rmason wrote: post #87
maxbond wrote: post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

Isn't that the same one that he was replying to?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#300

Post by _ea1k »

sclarisse wrote: post #217
katbyte wrote: post #95

Earlier quoted context omitted.

Blame the fact freight generally overrules passenger: https://youtu.be/qQTjLWIHN74 Amtrak isn’t allowed to be successful Either way rather be stuck on a train then a plane anyday

There’s a reason for that. Freight has priority because the freight companies actually own the rails, and actually use them, and it’s a significant contributor to the US economy. Amtrak is an unwelcome guest that they are required by law to put up with. If they wanted better service, Amtrak could pay for it, except the big cross country routes are already crazy unprofitable. The actual sane solution is to shut down t…

Doesn't Amtrak pay for that? I have a feeling if there was an effort to shut them down and replace it with buses, the freight companies would block it. They benefit from the relationship.
Post reply on HN