Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

371–380 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#371

Post by shuntress »

DonHopkins wrote: post #318
justinsaccount wrote: post #296

Earlier quoted context omitted.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

You act like "moving fast" is all you need to know to "move fast". As if it's simply the skill of making time move faster, and you don't need any other skills than that, because once you've broken the laws of physics and changed the speed of time, everything just works faster without any differences or consequences. Do you watch a lot of Superhero movies? You should work smarter, not harder. Just turn up your smart k…

Slow is smooth and smooth is fast.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#372

Post by eurasiantiger »

lsaferite wrote: post #192
eurasiantiger wrote: post #98

Earlier quoted context omitted.

Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.

Having dealt with utf8mb4 data being inserted into the utf8mb3 columns many many times in the past, I've never had a table "instantly go down". You either get silent truncation or a refusal to insert the data.

Well, your applications haven’t used a serialized or JSON column. That’s how you go from truncation to downtime.

That said, I do remember this being an issue even with plain text.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#373

Post by eurasiantiger »

exabrial wrote: post #306

Bunch of Node.js programmers lecturing the FAA on reliability is hilarious. Your framework is out of date, go use whatever the latest js framework published today is.

What’s even more hilarious is expert professionals being openly derisive and spreading stereotypes about others based on programming language preferences.

Do you hate JavaScript programmers?

Do you also agree with your friends how of people of different color are all the same?

Do you feel the need to call out other people’s sexual orientation, at least behind their back when only your friends are hearing?

Some of your friends might be JavaScript programmers, you know.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#374

Post by MichaelZuo »

jjk166 wrote: post #254
MichaelZuo wrote: post #140

Earlier quoted context omitted.

It sounds like it would be very simple to implement correctly if it was literally spelled out like what your describing?

The problem is that spelling out complicated things is hard. Take the law for example - in theory we have a coherent code that specifies exactly what things are crimes and the appropriate methods of dealing with them. In practice, it takes teams of highly trained professionals and an elaborate system of courts to clarify what these laws mean in all but the most trivial cases. Generally you need some flexibility to ha…

Which legal theory postulates a ‘coherent code that specifies exactly what things are crimes and the appropriate methods of dealing with them’?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#375

Post by MichaelZuo »

tominous wrote: post #163
MichaelZuo wrote: post #140

Earlier quoted context omitted.

It sounds like it would be very simple to implement correctly if it was literally spelled out like what your describing?

I just spent a month doing an E-Business Suite platform migration and it was very similar: follow the step-by-step instructions to apply patches and run commands. Each patch has a README file with dependent patches or commands which need to be completed first. It works mind-numbingly great until you run into the first of many circular dependencies. That's one problem with treating the implementer as a machine to run…

There can be multiple folks doing it in parallel and checking each other’s work, unless I’ve misunderstood?

If problems of circularity arise they can try to clarify the issue, or go back to the drawing board.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#376

Post by sokoloff »

__MatrixMan__ wrote: post #366
sokoloff wrote: post #339

Earlier quoted context omitted.

There are just under 3 M federal (civilian) employees. I think it's entirely unreasonable to think that we would pay a UBI to ~210 M adult citizens (a 70x multiple) at levels that would represent a greater amount of purchasing power than to the people nominally working for the federal government. If you're firm in your view that that's reasonable, I'd like to learn more about the proposal as to how the math would wor…

OK. I'm not going to code up a simulation because the research hasn't been done to confirm my choice of constants, but I can sketch it. Each workday is a function of the macroeconomic climate and some set of cultural norms during which we exhibit some blend of the following personae. As we'll see, introducing UBI reduces the prevalence of the bureaucrat persona which has knock-on effects leading to surplus. --- The M…

Where does the UBI money come from in this system, particularly if the surplus gets so bit that there's no reason to take a government job?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#377

Post by jupp0r »

Helmut10001 wrote: post #108
jupp0r wrote: post #37

Earlier quoted context omitted.

One more reason to use ZFS.

Surprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.

Not a lot of people are familiar with how well ZFS is engineered compared to most other file systems and how life changingly different it is.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#378

Post by haswell »

wbl wrote: post #332
haswell wrote: post #314

Earlier quoted context omitted.

This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”. This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down,…

Those were the wrong procedures. If you were regularly rebooting systems left and right you'd learn quickly if things didn't come up.

It’s always easy to say that in hindsight. But keep in mind this is an environment with many core components built in the 80s. Regular reboots on old AIX systems wasn’t a common practice - the sheer uptime capability of these systems was a big selling point in an environment that looks nothing like a modern cloud architecture.

But none of that is really the point. The point is that even with every correct procedure in place, you’ll still encounter failures.

Modern dev teams in companies that build software have more checks and balances in place from the get go that help head off some categories of failure.

But when an organization is built on core tech born of the 80s/90s, there will always be dragons, regardless of the current active policies and procedures.

The problem is that the cost to replace some of these systems was inestimable.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#379

Post by justinsaccount »

cloverich wrote: post #361
justinsaccount wrote: post #296

Earlier quoted context omitted.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

Change is the most common reason for breaking things. Moving fast means more broken things, hence the slogan. The alternative is indeed move slow, break things less often. It's a bad strategy when you NEED a LOT of change. But if you don't NEED a LOT of change, and you do need a lot of stability, it seems perfectly valid?

So for the things that don't need a lot of change, what are some characteristics of the system?

Does CI/CD exist? Does CI even exist? Are deployments automated? Is data sanity checked before loading? Is there a development environment? Do things like hourly snapshots exist? Can you easily provision a replacement system from scratch and restore data from a known good snapshot?

Or, is every process manual, slow, and error prone because there's never been a need to move fast.

Look at this one sentence in the article:

> In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system -- a significant decision, because the reboot can take about 90 minutes, according to the source.

So they do a reboot, that takes 90 minutes for some reason, and then that didn't even fix the problem. Their system that needs a lot of stability is now broken.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#380

Post by justinsaccount »

arcbyte wrote: post #312
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

The only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.

Yep, this is the exact point I tried to make above and got heavily downvoted.

If you can't move fast when things are working well, you can't move fast when things are broken. Acting like moving slow is going to prevent things from ever breaking is just wishful thinking.

Post reply on HN