Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

101–110 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#101

Earlier quoted context omitted.

[flagged]

WTF are you putting on your eggs that anyone can smell from more than a foot away? Seriously- a soft boiled egg (hard boiled more likely given the travel context here) is almost completely odorless…

It’s common to over-boil eggs that results with a green copper sulfate patina on yolk. Some people vehemently prefer that. We had some friends visit us and they took that on a plane… Yikes!

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#102
post #89

Earlier quoted context omitted.

WTF are you putting on your eggs that anyone can smell from more than a foot away? Seriously- a soft boiled egg (hard boiled more likely given the travel context here) is almost completely odorless…

They do have an odor to them, especially since most people tend to boil them to death but I can think of a hundred things more offensive in a confined space.

Boss used to microwave smoked mackerel in office kitchenette. Would coincide with half the staff leaving for lunch out :D

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#103
post #90

Earlier quoted context omitted.

What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. N…

This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…

My intuition on that is that two is also a bad number to choose in that case. One could fo full lunar mission on the thing and have three models and in case of inconsistency the majority wins.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#104

Earlier quoted context omitted.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.

Surely nobody would use that format and allow a commit message including emojis to cause an effective DOS for a large Sonarqube project.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#105
post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening.

imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody forms and keys and whatever password rotations etc etc. follow this process and fill out this presentation template wait 3 months and then present it to a board who doesn't give a shit or have any understanding of your project.

it's all an insane amount of work and it gets us the opposite in terms of the goals these processes are intended to achieve.

I have zero doubt that the only thing that will happen as a result of this catastrophic incident is another 50 lines in a spreadsheet somewhere telling you to do stuff that nobody will comprehend or implement correctly or even verify until there is a similar incident causing an investigation into it.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#106
post #97
post #85

Earlier quoted context omitted.

Probably he’s been to… any airport. Maybe they’re bearable if you can get into a lounge

More or less. The chairs are hostile to stop you from getting too comfortable. I've had 6 hour layovers before and had to sleep in those terrible chairs under fluorescent lights. On the last flight I took there was barely any good near my gate, there were like 4 bars instead. You can't leave your luggage anywhere, so you're dragging it all over the place. You get jammed like sardines into the plane, and you can't str…

> I will say the views are unbeatable

It's hard to agree with that. There are some amazing train routes through Austria, Norway, Switzerland, Germany that keep me looking out of the window the whole way.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#107
post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.

It could also be corruption caused by log-based database replication

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#108
post #37

Earlier quoted context omitted.

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

One more reason to use ZFS.

Surprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#109
post #89

Earlier quoted context omitted.

WTF are you putting on your eggs that anyone can smell from more than a foot away? Seriously- a soft boiled egg (hard boiled more likely given the travel context here) is almost completely odorless…

They do have an odor to them, especially since most people tend to boil them to death but I can think of a hundred things more offensive in a confined space.

> They do have an odor to them

I never had an egg that smelled unless it was bad (which too hasn't actually happened to me). My sense of smell is pretty good, I smell all kinds of little things (incidentally, I think that loss of sense of smell can be an early signal for approaching death, if there is no non-deadly cause to explain it).

I read the replies here: https://www.quora.com/Why-do-boiled-eggs-stink-but-scrambled... -- but I've never seen this in five decades of egg eating. I think I would remember a stinking egg (would make me more reluctant to make more for quite some time)? What does "overcook" mean in this context? I've done up to a little under 10 minutes for some larger eggs.

I think there may be something very wrong with the food, and subsequently the bodies, of your mass-incercerated and terribly treated chickens. I did see some funny eggs (inside colors) - that UI refused to eat - in places that bought the cheap mass-produced eggs. My mother also gets eggs with unnaturally deep colors, I think that's something they must have added to the chicken's prison food.

That's an issue I have with discussions - but also with studies - about this or that food, for example "meat is bad", but anything really, tomatoes or potatoes too. Each food type has a vast spread, and everybody just uses the name of the food but people are all talking past one another because they have vastly different actual food in mind.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#110
post #97

Earlier quoted context omitted.

More or less. The chairs are hostile to stop you from getting too comfortable. I've had 6 hour layovers before and had to sleep in those terrible chairs under fluorescent lights. On the last flight I took there was barely any good near my gate, there were like 4 bars instead. You can't leave your luggage anywhere, so you're dragging it all over the place. You get jammed like sardines into the plane, and you can't str…

> I will say the views are unbeatable It's hard to agree with that. There are some amazing train routes through Austria, Norway, Switzerland, Germany that keep me looking out of the window the whole way.

I love the views from the train too, but I feel like the view flying is something magical everyone should experience at least once in their life.

The ground fading away, the tiny cars, glittering rivers, seeing it in reverse for landing. That is something I'll wax poetic about. But if I could take the train the rest of my life I would.

Clearly the middle ground here is to bring back zeppelins. /s

Post reply on HN