Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

211–220 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#211

Post by VLM »

__MatrixMan__ wrote: post #165
mobilefriendly wrote: post #154

Earlier quoted context omitted.

This is by design, it meets the primary goal of the government which is funding patronage jobs that make few, if any, demands on the worker.

People who want a job like that should just be on UBI instead. Then at least we'd have systems that could change to meet the needs of their users in a timely way.

The people who need a pyramid structure to strive for, and office politics to fight, will never settle for "from each according to their ability and to each according to their need", and those are the people we've selected for.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#212

Post by metacritic12 »

avar wrote: post #209
metacritic12 wrote: post #196

Earlier quoted context omitted.

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

We know what it's like in software because we have e.g. the example of how the Space Shuttle's flight control software was developed and audited. It's weeks of meetings, tests and change processes etc. obsessing over changing a single instruction in assembly. Which also suggests to me that the problem isn't per-se the glacial progress & approval processes, but that they're using the wrong glacial processes. If this F…

I would settle for the government moving bureaucracy from safety-theatre to actual safety.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#213

Post by lebuffon »

marrone12 wrote: post #69
mgsouth wrote: post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

The CBC is great. I watch the news show every night even as an American.

To paraphrase Sarah Palin: Canada can see the USA from their front window. (A great deal of time is spent looking through that window)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#214

Post by Someone1234 »

JCM9 wrote: post #200
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code.

Obviously the NOTAM system isn't up to scratch, that's why we're talking about it. But "hard problems are easy" isn't constructive nor very realistic. I bet someone could put in a three-hour presentation talking about all the complexity that led us to this conclusion.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#215

Post by Someone1234 »

VLM wrote: post #206

"The source said the NOTAM system is an example of aging infrastructure due for an overhaul." The concept of "lint" a file and the concept of verifying a backup are truly ancient and coincidentally are also completely absent from the description of the problem. People that felt no need to do either 20 years ago are certainly not going to start doing it today, especially when the inevitable system failure in the dista…

I think you're misunderstanding what the "backup" was in this context (in particular as the article is unhelpful by calling it a backup file, which isn't accurate).

In this case it is two PRODUCTION systems running concurrently. The primary and the secondary (article calls the "backup"). Primary went down due to corruption, but the identical secondary system couldn't be switched to because the corruption also occurred there.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#216

Post by VWWHFSfQ »

Someone1234 wrote: post #215
VLM wrote: post #206

"The source said the NOTAM system is an example of aging infrastructure due for an overhaul." The concept of "lint" a file and the concept of verifying a backup are truly ancient and coincidentally are also completely absent from the description of the problem. People that felt no need to do either 20 years ago are certainly not going to start doing it today, especially when the inevitable system failure in the dista…

I think you're misunderstanding what the "backup" was in this context (in particular as the article is unhelpful by calling it a backup file, which isn't accurate). In this case it is two PRODUCTION systems running concurrently. The primary and the secondary (article calls the "backup"). Primary went down due to corruption, but the identical secondary system couldn't be switched to because the corruption also occurre…

So the backup system failed

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#217

Post by sclarisse »

katbyte wrote: post #95
rmason wrote: post #87

Earlier quoted context omitted.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

Blame the fact freight generally overrules passenger: https://youtu.be/qQTjLWIHN74 Amtrak isn’t allowed to be successful Either way rather be stuck on a train then a plane anyday

There’s a reason for that. Freight has priority because the freight companies actually own the rails, and actually use them, and it’s a significant contributor to the US economy. Amtrak is an unwelcome guest that they are required by law to put up with. If they wanted better service, Amtrak could pay for it, except the big cross country routes are already crazy unprofitable.

The actual sane solution is to shut down those routes and run buses. American heavy rail outside the NEC is better suited for freight anyway. Build new dedicated corridors for fast passenger services, or stay home.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#218

Post by bob1029 »

jve wrote: post #185

From the article and comments I don't understand - are they talking about backup (copy of data) or backup system (nuke your primary system, failover to backup system and just keep working) ? I think it is a SYSTEM > It has a backup, which officials switched to when problems with the main system emerged, according to the source. > Officials ultimately found a corrupt file in the main NOTAM system, the source told CNN.…

I think you might be right.

I am starting to wonder if this "backup" is an online log replica of the production system.

Failover doesn't work out in the situation where you are replicating trash. A "reboot" from an actual backup/snapshot would be required if you ate a bad log stream.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#219

Post by huslage »

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be minimized in any way. This is not some government corruption thing. There are hundreds of thousands of lives at stake every day.

I, for one, appreciate the work that the FAA does to keep us safe in the air and appreciate that this was handled appropriately. Everything breaks. It's just a matter of how and when and what we do with it when it does. The FAA handled this outage appropriately and in a timely manner. I feel for the engineers who had to work on this incident.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#220

Post by greggarious »

huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

If they’re so risk averse, why are their systems failing so severely?

It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

Post reply on HN