Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

341–350 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#341

Earlier quoted context omitted.

Which is why we've not had a commercial plane crash since.......2009? Seems FAA is pretty effective at reducing risk.

I haven't had a car crash, should I also thank the FAA?

If the FAA was involved in car safety and there was a reduction in car crashes, absolutely.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#342
post #301

Earlier quoted context omitted.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

That's just not true. Moving faster does not mean moving more effectively. Frantic activity should not be mistaken for progress. I'm OK with the FAA taking "days to resolve the problem" if it means that nobody dies. For what it's worth, my team and I are partial to "move carefully, and tend things".

"Moving faster" doesn't mean "frantic" either. Who said frantic?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#343
post #332

Earlier quoted context omitted.

This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”. This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down,…

Those were the wrong procedures. If you were regularly rebooting systems left and right you'd learn quickly if things didn't come up.

And then you find an obscure service doesn’t come back up on the 10,000th or 100,000th reboot because of . And now you have multiple states, because you have to handle failover. It’s turtles all the way down.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#344
post #278

Earlier quoted context omitted.

> 1) mostly thank the airline operators for that. This is a 100% proof that you have not the faintest clue what you're talking about. The #1 priotity of airline operators is profit, not safety.

Another way to think about it is that safety is good for profits, at least in so far as consumers care about it. And I think consumers do care quite a bit. Of course there are arguments to be made for regulation, but I think your statement is overly emphatic.

> And I think consumers do care quite a bit.

In theory, if you ask explicitly, of course they do. In practice, they choose the cheapest ticket and maybe avoid airlines that have had a high profile incident recently. They don't have the time or means to actually evaluate an airline's safety culture.

Meanwhile, executive decisions are driven by quarterly earnings reports, and you can cut a lot of corners for quite a number of quarters before your luck runs out and 200 people die.

My statement is, if anything, not emphatic enough.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#345
post #4

> In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system -- a significant decision, because the reboot can take about 90 minutes, according to the source. 90 minutes; this certainly appears to be an advertisement about Windows OS.

Or just a legacy UNIX with a lot of disks. I've seen PH-UX take over an hour to boot due to that.

(Just to be clear I meant specifically just enumerating SCSI LUNs, not fsck or anything like that)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#346

Earlier quoted context omitted.

Which is why we've not had a commercial plane crash since.......2009? Seems FAA is pretty effective at reducing risk.

a few things 1) mostly thank the airline operators for that. 2) the fact the 737 Max disasters happened outside the US was a matter of chance not exemplary policies by the FAA. FAA policies did not prevent the deployment and roll out of this dangerous aircraft in the United States and did not lead to grounding of the aircraft until multiple events occurred. 3) I'm mostly talking about software here, and I believe a b…

[deleted]

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#347

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

FAA regulations, more so than nearly anything else regulated in the US, are written in blood. They are not written out of government corruption. Many many many hundreds of people have died in airplane accidents since airplanes were invented, and the reason why your average person can board a commercial flight and act like they just got on the bus (with a /better/ risk profile than a bus on public roads) is exactly due to these regulations. The FAA has and continues to lead the entire world in how to do reasonable and meaningful air traffic regulation.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#348
post #340

Earlier quoted context omitted.

I was a software engineering manager at the FAA for more than six years.

That doesn't make the statement any less ridiculous. What was the ultimate reason for the 737 MAX debacle? That airlines want to save money on type rating training. Look at accident reports, and half the time the airline's safety culture (or lack thereof) is at least a contributing cause. The FAA may be in many ways dysfunctional, but so are the airlines, and it's sure as hell not them who are pushing for better safe…

point was that if FAA software policies around safety were actually effective we would not have had the 737 max issues.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#349
post #271

Earlier quoted context omitted.

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

When I hear "move fast and break things", I am reminded of a guy that I worked with. He delivered work fast......full of bugs...0 planning...and his daily ritual was to just keep patching the "pile of shit" he put together.

He has now moved on and we sometimes chat, he works for a huge corp, still sucks at writing SQL.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#350

Earlier quoted context omitted.

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.

At my old firm, our federal project profit margin was a bit lower than median, though once you factor in the sales, contracting, and leg overhead the bottom line was somewhat worse than the project actuals made them look. Since our side of the house did relatively short-burn contracts (6-12 months), federal work was generally not that valuable for us; I used it for filler work when our usual sales pipeline was weak. The real value was for the side of the house that did long-term software and support work or heavy citizen support outsourcing, when contract durations can be measured in decades. Same for a friend who inked a $5bn DOD deal; the margin isn’t great, but it’s a 10-year deal that gives her a stable cash flow basis to grow on.

(Also, OP is probably underestimating the full-sheet cost of a federal FTE, as well as the complexities of fund-based budgeting and forecasting.)

Post reply on HN