Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

91–100 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#91
post #34

I find it quite weird multiple things failed on the same day, such as the UK post office system unable to deliver abroad

I suspect this is just statistics, like "it is likely that two people will have the same birthday in a roomful of people"

Would this be an example of the Poisson distribution? (Perhaps exaggerated by post holiday upgrades?)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#92
post #87
post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

That's terrible. On the other hand, airports for me are guaranteed to be awful, and I've always enjoyed the Amtrak. If I'm in a hurry to get someplace I'll take a plane, sure. I wish we had high speed rail in this country (or I'd never fly again except internationally) but of course we don't. If there's no particular rush I'll take my sweet time on the Amtrak, and if it's delayed for 30 hours - like I said, you'll find me minding my reading with no complaints.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#93
90 minutes boot time...

I have a feeling the summarized age of the hard drives in that system exceeds the age of the United States itself (which is 247 years). When fsck was introduced in 4BSD in 1980, it checked every filesystem on boot because it had no better idea. If this thing is, say, forty years old then that's exactly the right age for this...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#94

Earlier quoted context omitted.

Well, sometimes it's because you already boarded and started to taxi and there are so many planes on the ground that there are no gates open to get back to the terminal so you spend hours on the plane waiting for a slot to open so you can get off, meanwhile the galley runs out of food/drink and the toilets fill up. Then when you finally get into the terminal, it's chaos, no one knows when planes will be flying again…

Southwest?

YVR a month ago

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#95
post #87
post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

Blame the fact freight generally overrules passenger: https://youtu.be/qQTjLWIHN74

Amtrak isn’t allowed to be successful

Either way rather be stuck on a train then a plane anyday

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#96
post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

Someone put an emoji in a NOTAM

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#97
post #85

Earlier quoted context omitted.

What is your preference based on?

Probably he’s been to… any airport. Maybe they’re bearable if you can get into a lounge

More or less. The chairs are hostile to stop you from getting too comfortable. I've had 6 hour layovers before and had to sleep in those terrible chairs under fluorescent lights. On the last flight I took there was barely any good near my gate, there were like 4 bars instead. You can't leave your luggage anywhere, so you're dragging it all over the place. You get jammed like sardines into the plane, and you can't stretch your legs. You can't really relax because you have to stay abreast of announcements, in case they change your gate or cancel your flight or what have you. Everyone is on edge. Et cetera.

On the train there's more room, you can move around, and the atmosphere is more relaxed.

Some people like planes. I saw an HN comment where someone said they took lots of "flights to nowhere" over COVID and used the plane as their office. I don't understand that at all. But to each their own.

I will say the views are unbeatable.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#98

Earlier quoted context omitted.

> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#99
post #90

Earlier quoted context omitted.

What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. N…

This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…

you could go to a model that verifies the integrity of the data coming in and makes sure the limits on the data are sane before committing it to the db. using a language with strong safety principles (that are not very "hip") like ada or fortran. or you design it so that the system is robust to failure and expects failure like something from teleco like erlang. redundant hardware is fine and great but having them do verification on the data and monitoring the limits of the system is pretty important too.
Post reply on HN