Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

331–340 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#331

Post by zhengyi13 »

bastardoperator wrote: post #282
alexb_ wrote: post #271

Earlier quoted context omitted.

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

I read this incident as they in fact did have a DR strategy, they knew how to execute it, did so successfully, and still failed.

The only thing I'm left wondering at this point is whether the corrupt data were a config, or state.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#332

Post by wbl »

haswell wrote: post #314
bastardoperator wrote: post #298

Earlier quoted context omitted.

We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.

This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”. This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down,…

Those were the wrong procedures. If you were regularly rebooting systems left and right you'd learn quickly if things didn't come up.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#333

Post by icedchai »

f1refly wrote: post #118
pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

It doesn't. All those unit tests don't guarantee you have a working system, especially when everything is mocked out (which is often the case.)

End-to-end and integration tests are much more helpful. But even then, they won't look at operational concerns like backup and recovery.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#334

Post by icedchai »

tyingq wrote: post #280
defrost wrote: post #117

Earlier quoted context omitted.

Given the vintage pedigree BSD like SunOS is an even better candidate. Once tuned for processing they just ran - rip out a network cable and stuff it back in again, not a hiccup. Many of the time saw no benefit in porting forward to Slow Loris.

I don't think weblogic exists for SunOS.

Correct. Java never ran on SunOS 4.x (aka Solaris 1.x), only Solaris 2.x (aka SunOS 5.x. I loved Sun's naming schemes.)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#335

Post by exabrial »

czbond wrote: post #311
exabrial wrote: post #306

Bunch of Node.js programmers lecturing the FAA on reliability is hilarious. Your framework is out of date, go use whatever the latest js framework published today is.

I'm sorry you can't fly today because the NOTAM system sent NaN alerts.

Not An airplaNe

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#336

Post by justinsaccount »

DonHopkins wrote: post #318
justinsaccount wrote: post #296

Earlier quoted context omitted.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

You act like "moving fast" is all you need to know to "move fast". As if it's simply the skill of making time move faster, and you don't need any other skills than that, because once you've broken the laws of physics and changed the speed of time, everything just works faster without any differences or consequences. Do you watch a lot of Superhero movies? You should work smarter, not harder. Just turn up your smart k…

I'm not sure what point you are trying to make. Physics? super heroes? knobs?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#337

Post by guenthert »

ilyt wrote: post #181
mdavidn wrote: post #51

Earlier quoted context omitted.

The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.

Neither checks the checksum on every read as that would be performance-prohibitive. So "bad data on drive -> db does something with corrupted data and saves corrupted transformation back to disk" is very much possible, just extremely unlikely. But they said nothing about it being bad drive, just corrupted data file, which very well might be software bug or operator error

> Neither checks the checksum on every read as that would be performance-prohibitive.

It is expensive. It might be prohibitive in a very competitive environment. This is hardly the case here. Safety first!

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#338

Post by rprospero »

nonethewiser wrote: post #242
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

What produces this sort of environment? Is it basically a way for everyone to shed responsibility? I guess even that wouldnt answer it, since this isnt really a problem in private industry.

In my experience, this arises as an unintended consequence of the quest to lower costs and reduce bureaucracy.

About ten years ago, a new manager was brought in to make us act less like a moribund government department and behave more efficiently. As an example of government waste, he pointed to the money we were spending on storage for data back-ups. We wouldn't need back-ups if we stopped making mistakes.

You might think that this no-back-ups policy would be an instant disaster, but it lasted years without issue. When there was a failure, the manager would hand the sys-admin a soldering iron, the admin would fix the hard drive, and we would be back on track. Finally, the sys-admin retired and a new one replaced him. Not long after, a critical system failed and data that we were required by law to maintain was lost. The manager handed the admin a soldering iron and told him to fix the hard drive. The admin said it was impossible and the manager fired him (yes, you can get fired from a government job). Other candidates were interviewed, but no one applying for a $30k job was confident that they could repair a broken hard drive.

Finally, there was talking of hiring the old admin to come out of retirement and fix the drive. Except he explained that it had always been impossible. During his tenure, he'd spent 5% of his salary (gross, not net) paying for back-ups and replacement drives. When the manager gave him a soldering iron, he'd just chuck out the old drive, by a replacement off Newegg with his personal credit card, and load it with the data he'd backed up to his personal S3 storage. His back-up script was still running on the server, but he'd stopped paying for the storage space the moment he retired.

Eventually, the manager was forced to spend a whole year's budget on an expensive data-retrieval firm to collect the data (which was still cost an order of magnitude less than the fine the department would have had to pay if we'd lost the data). He was fired and a new manager brought on board. Because of the money which had been lost on the data retrieval, new measures were put in place to prevent this from ever happening again. This included a new back-up system and audits to ensure that other employees were using personal funds to pay for departmental resources. Of course, this meant rigorously documenting exactly what resources each employee was using...

Six years after the manager was brought in to decrease cost and increase agility, we were now more over budget and tightly controlled than we'd ever been.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#339

Post by sokoloff »

__MatrixMan__ wrote: post #319
sokoloff wrote: post #168

Earlier quoted context omitted.

UBI will never pay what a government job pays in purchasing power; that part is just math I think.

I don't think that is unreasonable to expect that government pay in a world where the government is a welfare program with a governing hobby might have less purchasing power than UBI in a world where we prioritize effective governance over beaurocracy. It's not a zero sum game.

There are just under 3 M federal (civilian) employees. I think it's entirely unreasonable to think that we would pay a UBI to ~210 M adult citizens (a 70x multiple) at levels that would represent a greater amount of purchasing power than to the people nominally working for the federal government.

If you're firm in your view that that's reasonable, I'd like to learn more about the proposal as to how the math would work.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#340

Post by brazzy »

gnarbarian wrote: post #297
brazzy wrote: post #278

Earlier quoted context omitted.

> 1) mostly thank the airline operators for that. This is a 100% proof that you have not the faintest clue what you're talking about. The #1 priotity of airline operators is profit, not safety.

I was a software engineering manager at the FAA for more than six years.

That doesn't make the statement any less ridiculous.

What was the ultimate reason for the 737 MAX debacle? That airlines want to save money on type rating training.

Look at accident reports, and half the time the airline's safety culture (or lack thereof) is at least a contributing cause.

The FAA may be in many ways dysfunctional, but so are the airlines, and it's sure as hell not them who are pushing for better safety standards, it's the FAA and (especially) the NTSB.

Post reply on HN