Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

381–390 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#381

Post by __MatrixMan__ »

sokoloff wrote: post #376
__MatrixMan__ wrote: post #366

Earlier quoted context omitted.

OK. I'm not going to code up a simulation because the research hasn't been done to confirm my choice of constants, but I can sketch it. Each workday is a function of the macroeconomic climate and some set of cultural norms during which we exhibit some blend of the following personae. As we'll see, introducing UBI reduces the prevalence of the bureaucrat persona which has knock-on effects leading to surplus. --- The M…

Where does the UBI money come from in this system, particularly if the surplus gets so bit that there's no reason to take a government job?

We issue it to ourselves, more or less like CirclesUBI is doing it in Berlin.

They're just letting it be inflationary and setting the payout to increase over time to adjust for inflation. So maybe you get $5 per week this year and $8 per week next year... This can be balanced so that it amounts to a more or less constant purchasing power.

Personally I prefer the demurrage approach where account balances just have a decay rate--that way you've got a better shot at $5 written down today having the same meaning to people who read it next year, but the economics are the same (more on the theory here: http://en.trm.creationmonetaire.info/ ).

It's gotta be decoupled from the government so that, as discussed in my model, it can act as a safety net while you're ridding yourself of wasteful bureaucracy. It doesn't really work if the bureaucrat you're deposing can threaten to take away your UBI.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#382

Post by landemva »

haswell wrote: post #290
bastardoperator wrote: post #282

Earlier quoted context omitted.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…

> incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident

I'm not surprised. FAA does not fly each plane. Government organizational complexity helps ensure the government organization survives through next round of Congressional appropriations.

Org complexity + opaque oversight + 'safety' + 'homeland security' + taxpayer funded = playing around and more budget.

The pilot is responsible for safety. Air travel has rules to avoid collisions (eastbound gets altitude levels different than westbound, pilots shall broadcast on known frequencies) and pilots have distributed intelligence to keep their flight safe.

Yes, somehow there needs to be coordination of runway use. Many ways to provide reservations and queuing.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#383

Post by krabizzwainch »

zhengyi13 wrote: post #331
bastardoperator wrote: post #282

Earlier quoted context omitted.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

I read this incident as they in fact did have a DR strategy, they knew how to execute it, did so successfully, and still failed. The only thing I'm left wondering at this point is whether the corrupt data were a config, or state.

I don't even know if I'd call this a disaster recovery fail. Depending on what they meant by "corruption", a roughly 6-8 hour turn around time is not awful for a database restore.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#384

Post by katbyte »

sclarisse wrote: post #217
katbyte wrote: post #95

Earlier quoted context omitted.

Blame the fact freight generally overrules passenger: https://youtu.be/qQTjLWIHN74 Amtrak isn’t allowed to be successful Either way rather be stuck on a train then a plane anyday

There’s a reason for that. Freight has priority because the freight companies actually own the rails, and actually use them, and it’s a significant contributor to the US economy. Amtrak is an unwelcome guest that they are required by law to put up with. If they wanted better service, Amtrak could pay for it, except the big cross country routes are already crazy unprofitable. The actual sane solution is to shut down t…

> Build new dedicated corridors for fast passenger services

If only, one of the best investments for NA would be an actual HSR network outside freight. California is trying and there are so many people who are doing everything they can to stop it. Including musk inventing an impossible alternative he never invented to build[1], hyperloop, as HSR would compete with tesla and hurt his sales.

[1] https://www.fresnobee.com/opinion/editorials/article26445107...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#385

Post by nucleardog »

arcbyte wrote: post #312
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

The only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.

> Instead, if you can move fast consistently, you can minimize the total downtime.

In safety critical software where _a_ failure can result in loss of life, is “total cumulative duration of downtime” really the metric we’re optimizing for?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#386

Post by birdyrooster »

tenken wrote: post #66
birdyrooster wrote: post #50

Earlier quoted context omitted.

Imagine you have one node which is running as a replica of another and it takes the backups. Well, let’s pretend it is backing up the corrupted data once in a while and it happened to overwrite their cold backup. They could have any number of databases and still had this failure. It’s more their methodology for taking backups. They should have many points in time to choose from to rebuild their database. They should…

> They should be testing their databases before backing them up blindly. Oh you mean they should be testing/validating the generated backup db file before replicating it to long-term archive ...

semantics but yes

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#387

Post by simonh »

arcbyte wrote: post #312
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

The only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.

Downtime isn’t the metric their procedures are optimised to minimise. It’s optimised to minimise air traffic accidents. Moving fast might minimise total down time (though I seriously doubt that), but what effect would it have on accuracy and reliability? Mistakes mean dead people. In this incident zero people died. You really sure you know you can improve on that?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#388

Post by learningstud »

avgDev wrote: post #358
justinsaccount wrote: post #356

Earlier quoted context omitted.

And as I said, people think the alternative to "fast, full of bugs, 0 planning" is automatically "slow, no bugs, lots of planning" but it's often "slow, lots of planning, just as many bugs"

When you are in a complex spiderweb you simply cannot move fast. If you are moving fast you are not looking at everything and it will blow up in your face.

Or, you can have complex systems with PROOFS like CompCert and Sel4. With formal methods, you can move as fast or as slow as you like.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#389

Post by eli »

nonethewiser wrote: post #239
eli wrote: post #234

Earlier quoted context omitted.

They don't completely eliminate risk, you mean?

He quite clearly does not mean that. Please address his perspective in good faith.

A bad outcome here doesn’t demonstrate their policies are ineffective any more than a car crash demonstrates that seatbelts are ineffective.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#390

Post by haswell »

justinsaccount wrote: post #342
eckza wrote: post #301

Earlier quoted context omitted.

That's just not true. Moving faster does not mean moving more effectively. Frantic activity should not be mistaken for progress. I'm OK with the FAA taking "days to resolve the problem" if it means that nobody dies. For what it's worth, my team and I are partial to "move carefully, and tend things".

"Moving faster" doesn't mean "frantic" either. Who said frantic?

Frantic is often a natural outcome of “moving faster” when the environment one is moving in is not conducive to that speed of movement.

In my experience, this tendency towards frantic is multiplied the larger and more complex the organization and architecture becomes.

The entire point of “move fast” in software circles is to leave behind the constraints of legacy tech and management practices in favor of building something “better”.

In a mature org that grew up before these ideas were mainstream, maybe one or two teams can manage to move faster, but invariably they end up depending on other teams, who in turn depend on deeply ingrained and established company culture and procedures.

We can talk about why those impediments are a Bad Thing, and I wouldn’t advise a consumer startup to adopt those methodologies in 2022, but there’s still the harsh reality that where they exist, “just move faster” doesn’t help much more than telling a depressed person to “just do cardio every day”. There’s often a lot of inner work that’s gotta happen to make way for the new.

The only way I’ve seen this sort of work in a large org is when a brand new “emerging tech” group is spun up and given autonomy to work outside of the legacy norms. This is not perfect either, and seems much better for greenfield projects. When applied to deeply entrenched legacy systems, all of the problems mentioned above come to a head.

This also creates a weird in/out group dynamic which tends to further stratify the old tech and widen the gap between the old practices and the new.

In the context of this particular conversation though, I think the concept of “move fast” has lost all meaning and has little to offer for an org like the FAA.

Post reply on HN