Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

221–230 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#221

Post by chasd00 »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

yeah working for government is strange. I'm in a position where the product owners don't want to make any enhancements/changes to a production system out of budget concerns. However, the actual invoice for my team's time is the same whether they make changes or not. They don't want to "spend the money on enhancements to a functioning system" but the invoice amount is the same every month regardless.

oh and in my experience, government fte's can screw up an infinite amount of times with no risk to their job.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#222

Post by greggarious »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

>Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired

Weird, my impression was federal employment is the opposite: mess up as much as you want and you’ll be retrained and reassigned.

Or a that only in contexts like harassment or drug use?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#223

Post by Cthulhu_ »

greggarious wrote: post #220
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

Maybe the better question to ask is how bad would it be if they *weren't* so risk-averse.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#224

Post by gnarbarian »

huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

demonstrably the policies employed by the FAA do not effectively avert risk (in software). It's more akin to risk aversion theater.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#225

Post by Cthulhu_ »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

It sounds less like risk averse and more like cover-your-ass though; go through long checklists and committees and people so that no one person can be held responsible for any problems.

The closest thing you'll see in tech will be at back-end software like banks, insurance companies, pension funds, investment companies, embedded engineering / SCADA, ERP systems, etc. The "move fast and break things" mindset seems to mainly be a thing in Silicon Valley internet companies / start-ups, and mainly the latter because they're driven to pump up their own value instead of provide reliable software. Because if Twitter is down, it's an inconvenience, but if airplane software fails, lives are on the line.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#226

Post by tomrod »

metacritic12 wrote: post #196
gnarbarian wrote: post #105

Earlier quoted context omitted.

I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

> Agencies like the FAA are notoriously risk adverse.

There is a difference from risk averse ("we require heavy testing before deployment") and dysfunctional ("we are terrified of breaking anything but are unwilling to invest in maintenance").

Take the Air Force. Their risk decision is: can we accomplish a specific mission at hand with ideally minimal loss of warfigher life. If they don't maintain their planes, they cannot achieve the warfighter life loss minimization.

Similar with IT generally. Kicking the can down the road just grows the problem.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#227

Post by metacritic12 »

greggarious wrote: post #222
metacritic12 wrote: post #196

Earlier quoted context omitted.

Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety"). We all know this from the subject the FAA regulat…

>Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired Weird, my impression was federal employment is the opposite: mess up as much as you want and you’ll be retrained and reassigned. Or a that only in contexts like harassment or drug use?

You're right, my "mess up once, get fired", I really mean if a flight crashes and the FAA could have done something, they'll get political flak. High level political appointees may need to resign. Whereas if an agency does nothing for a term, the political leaders get to stay.

For routine FTEs, as a sibling post mentioned, you can probably mess up in a lot of ways (short of murder or being really non-PC) and still keep your job.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#228

Post by malfist »

gnarbarian wrote: post #224
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

demonstrably the policies employed by the FAA do not effectively avert risk (in software). It's more akin to risk aversion theater.

Which is why we've not had a commercial plane crash since.......2009?

Seems FAA is pretty effective at reducing risk.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#230

Post by jjk166 »

Cthulhu_ wrote: post #223
greggarious wrote: post #220

Earlier quoted context omitted.

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

Maybe the better question to ask is how bad would it be if they *weren't* so risk-averse.

Imagine how bad it would be if Kim Jong Un *weren't* so pro-american. It's a ridiculous hypothetical. Everything can always be worse, that is not a testament to an entity's quality.
Post reply on HN