Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

241–250 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#241

Post by sumtechguy »

JCM9 wrote: post #200
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

There is a whole group of people who their job depends on the system existing like it does. One dude I worked with re-did a procurement system with a few spreadsheets. He got tired of waiting for 30 people to 'do it'. As soon as his boss found it the whole project he was on was scrapped and he was demoted shortly thereafter. His 'sin'? He had accidently found a way to put 30 people out of a job. These systems exist like this because our gov wants them that way. Not because they are the best.

Yesterday should have been 'run this on these 2 old boxes and make sure they still work' regression test. Sounds like either that step is not there, totally skipped, or does not match reality. Building a real five 9s type system means taking each piece and pondering 'what are the different ways this can fail'. Then mitigating each of those. It is mind numbing tedious work than takes a long time to do. Also most of this is probably run by contract houses. Which means the people who use it do not really 'own it'. Which is by design, for CYA. Which costs more and takes more time because it is all paperwork.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#242

Post by nonethewiser »

gnarbarian wrote: post #105
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

What produces this sort of environment? Is it basically a way for everyone to shed responsibility?

I guess even that wouldnt answer it, since this isnt really a problem in private industry.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#243

Post by __MatrixMan__ »

gte525u wrote: post #208
__MatrixMan__ wrote: post #167

It sounds like the kind of problem that blockchains were invented to prevent. Important systems should not blindly replicate data, they should validate it.

This is the opposite of a blockchain in that there are centralized, trusted authorities (FAA, EASA) distributing this info.

Ok so it wouldn't be a public blockchain. The point is that when you replicate data you reject it if it doesn't conform to some ruleset. That way data that would break the system fails to replicate, and the scope of the problem is limited to whoever (or whatever) entered the bad data.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#244

Post by rattlesnakedave »

Someone1234 wrote: post #214
JCM9 wrote: post #200

Earlier quoted context omitted.

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. Obviously the NOT…

> This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code.

This stuff is such cringe, it is the type of response you see from jaded boomers more focused on box-ticking and punching out as opposed to doing anything new.

This is a situation where tossing the whole damn thing out and starting over again would be productive. The systems that lead to the creation of these half-fossilized government projects (that still don't work!) will not change and needs to be tossed.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#245

Post by Waterluvian »

gnarbarian wrote: post #224
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

demonstrably the policies employed by the FAA do not effectively avert risk (in software). It's more akin to risk aversion theater.

Demonstrably flu vaccines do not effectively avert risk. Demonstrably auto safety standards do not avert risk.

But they do. And they've been so incredibly effective that we have collectively forgotten what risk used to feel like, so we're ready to say we don't need these standards/organizations or that they're not working. Obviously this is not to say they are perfect or free from criticism. But it's not just theatre.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#246

Post by nonethewiser »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

Seems like Space-X is a counterpoint to this type of operation. They are much more efficient and have been rather reliable.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#247

Post by PaulWaldman »

>In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system – a significant decision, because the reboot can take about 90 minutes, according to the source.

How often is the system rebooted? I haven't heard of it happening before and some quick searching didn't find any historical examples. Is it a scheduled event and no flights have departure times while this maintenance is taking place?

I've found that issues tend to manifest on production systems with infrequent restarts.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#248

Post by nradov »

avar wrote: post #209
metacritic12 wrote: post #196

Earlier quoted context omitted.

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

We know what it's like in software because we have e.g. the example of how the Space Shuttle's flight control software was developed and audited. It's weeks of meetings, tests and change processes etc. obsessing over changing a single instruction in assembly. Which also suggests to me that the problem isn't per-se the glacial progress & approval processes, but that they're using the wrong glacial processes. If this F…

The NASA Space Shuttle flight software was among the best ever developed. There was never a defect that impacted safety.

But that team had kind of an "unfair" advantage in that were able to program in assembly code on bare metal with no real software stack. Whereas the rest of us are forced to build on a foundation of sand using multiple layers of low-quality third-party software in order to deliver any useful functionality.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#249

Post by haswell »

rattlesnakedave wrote: post #233
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

> You all seem to think this is similar in value or operation to a web app This line of reasoning is pure, 100%, cope. There is a reason this was broken and grounded flights nationwide. Providing cover for systems that allowed this to happen is not productive.

> This line of reasoning is pure, 100%, cope.

What does this even mean?

There’s a meaningful difference between providing cover and reminding people that major systems like the one in question are in a different category than the average web app.

The fact that such a nationwide stoppage hasn’t occurred since 2001 speaks to the stability of these systems, and I’m not sure what you’re suggesting here.

> There is a reason this was broken and grounded flights nationwide. Providing cover for systems that allowed this to happen is not productive.

I’m not trying to sound snarky here, but things do tend to break for reasons. Are you suggesting that there is a cure?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#250

Post by ssklash »

VLM wrote: post #210
avidphantasm wrote: post #173

Earlier quoted context omitted.

All the federal employees I have met are EXCELLENT. It is very difficult to get hired into one of the dwindling jobs at the agencies. A lot of the government has been contracted out. Having worked at a federal IT contractor, I can say that in my experience most of those workers are very good and dedicated to their work. HOWEVER, they don’t always understand well the mission at hand. Some of this is to be expected giv…

Ironically extreme selection pressure is exactly what leads to extreme risk aversion. "We only hire the absolute best" does not lead to "move fast and break things" (which sounds awful in a FAA context anyway) it leads to people who devoted their lives to being the absolute best at coloring inside the lines and never straying off the path, to being the best follower out there, to the ultimate authoritarians desiring…

David Graeber?
Post reply on HN