Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

201–210 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#201
> NOTAMs (Notice to Air Missions)

OT, but didn't NOTAM use to mean Notice to Airmen? It's still that on the ICAO website: https://www.icao.int/safety/istars/pages/notams.aspx

Did it change, or is this a CNN initiative? (And how does one notify a mission?)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#202
post #179
post #103

Earlier quoted context omitted.

My intuition on that is that two is also a bad number to choose in that case. One could fo full lunar mission on the thing and have three models and in case of inconsistency the majority wins.

It is but it doesn't matter if it is the input message that crashes it. You just need 3 bad message (sooo a message and 2 retries) to crash it whole

or 1 message that crashes all three on the first try vecause you are voting.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#203
post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

Just my 2 cents: 2 billons are payed not for reliability but for size. Size of the system (and size of the team).

Reliability makes the system and team bigger.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#204
post #201

> NOTAMs (Notice to Air Missions) OT, but didn't NOTAM use to mean Notice to Airmen? It's still that on the ICAO website: https://www.icao.int/safety/istars/pages/notams.aspx Did it change, or is this a CNN initiative? (And how does one notify a mission?)

The original definition was deemed to be not inclusive by DEI so address that they kept the acronym but changed the meaning.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#205
Makes one think if they should have hourly snapshots combined with integrity checks of the snapshots (and the main database, assuming it is feasible to run that for an online database). They will know within an hour if a snapshot is corrupt + they can restore from the last known good snapshot and not lose too much data and be up relatively quickly. Obviously like most software, there is a bunch of "it depends" but would be interesting to know more about the system.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#206
"The source said the NOTAM system is an example of aging infrastructure due for an overhaul."

The concept of "lint" a file and the concept of verifying a backup are truly ancient and coincidentally are also completely absent from the description of the problem. People that felt no need to do either 20 years ago are certainly not going to start doing it today, especially when the inevitable system failure in the distant future results in yet another lucrative replacement contract in the distant future for system 3.0.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#207
post #200
post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

Well they claim to be "risk averse", the wisdom there seems to be that if you fill out a million forms that qualifies as risk averse because it shows you did due diligence. The problem is they wholesale took that process from the rest of the org and applied it to software which doesn't work.

It's like making a crap sandwich and filling out a bunch of forms proving that it's not crap sandwich, rather than just spending the time you would be filling out all those forms on i don't know...not making a crap sandwich.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#208

It sounds like the kind of problem that blockchains were invented to prevent. Important systems should not blindly replicate data, they should validate it.

This is the opposite of a blockchain in that there are centralized, trusted authorities (FAA, EASA) distributing this info.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#209

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

We know what it's like in software because we have e.g. the example of how the Space Shuttle's flight control software was developed and audited. It's weeks of meetings, tests and change processes etc. obsessing over changing a single instruction in assembly.

Which also suggests to me that the problem isn't per-se the glacial progress & approval processes, but that they're using the wrong glacial processes.

If this FAA software was developed with something like the Space Shuttle's process it would still take forever to change something, but at least you'd end up with a function that you could mathematically prove would be able to handle any conceivable input.

I think you're never going to convince the government of a "move fast" culture. Even if the overall cost of grounding planes would be less than the cost of more reliability they'd never go for it.

Too many people's asses are on the line, and they're not having to spend their own money. The only thing they have to "pay" for is possible loss of face, or loss of political capital, both of which can be insured against effectively for free with taxpayer dollars.

But you might just be able to convince them that they're using the wrong sort of bureaucracy. You'd still spend a billion or two on something that should cost a million, but at least you'd get actual reliability as a result.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#210

Earlier quoted context omitted.

This is by design, it meets the primary goal of the government which is funding patronage jobs that make few, if any, demands on the worker.

All the federal employees I have met are EXCELLENT. It is very difficult to get hired into one of the dwindling jobs at the agencies. A lot of the government has been contracted out. Having worked at a federal IT contractor, I can say that in my experience most of those workers are very good and dedicated to their work. HOWEVER, they don’t always understand well the mission at hand. Some of this is to be expected giv…

Ironically extreme selection pressure is exactly what leads to extreme risk aversion.

"We only hire the absolute best" does not lead to "move fast and break things" (which sounds awful in a FAA context anyway) it leads to people who devoted their lives to being the absolute best at coloring inside the lines and never straying off the path, to being the best follower out there, to the ultimate authoritarians desiring to grow into being the authority.

The heaviest selection pressure usually does not lead to the most efficient system, it generally leads to a system able to endure heavy selection pressure.

There's a sociologist who wrote a famous book about bureaucracy and its in my library at home and the name of the sociologist and his book are at the tip of my tongue but he wasn't near the top of a quick google search; the above is a paraphrase of his book. No its not Douglas Adams or even Scott Adams although those two are correct about the problem in general LOL.

Post reply on HN