How I Fired Myself • Hacker News Board

Re: How I Fired Myself

#331

Post by richo » Thu, Feb 28, 2013, 6:32 AM UTC

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

You're forgetting the guy who didn't speak up to say "Hey, maybe we shouldn't do this in prod?".

I feel for him, but at the same time there's a point at which you have to if testing guns by shooting them near (but not specifically at) your coworkers is actually a good idea.

Re: How I Fired Myself

#332

Post by bcbrown » Thu, Feb 28, 2013, 6:42 AM UTC

ghjm wrote: post #154

paganel wrote: post #140

Earlier quoted context omitted.

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

This is correct and is the way it should be. So how come the programmers are always politically gunning for the keys to the production server cabinet, where you do have to be afraid to break things?

That's where raises and bonuses come from.

Re: How I Fired Myself

#333

Post by misleading_name » Thu, Feb 28, 2013, 6:44 AM UTC

It's not your fault. it's the fault of the person who cancelled backups, the person who didn't check that backups are being created, the "senior" people who let you work on the production database.... etc.

Re: How I Fired Myself

#334

Post by ommunist » Thu, Feb 28, 2013, 7:01 AM UTC

You are accidental hero and should be proud! You freed thousands of souls from one of the worse digital addictions.

Re: How I Fired Myself

#335

Post by bwb » Thu, Feb 28, 2013, 7:04 AM UTC

It was a mistake, but not huge. You should never have not have had backups, and that wasn't your responsibility. + their should have been a dev instances and a proper coding environment.

So don't blame yourself there!

Re: How I Fired Myself

#336

Post by miahi » Thu, Feb 28, 2013, 7:18 AM UTC

jms wrote: post #301

ajross wrote: post #211

Earlier quoted context omitted.

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Back when I worked for a small ISP, we had a diesel generator in case the power went out longer than our UPS batteries would last. This provided a great sense of security until we decided to test the system by powering off the main break and... it didn't start! It turns out the emergency stop button was pushed in. Easy enough for us to fix then, but if the power had gone out at 4am it would have been quite another ma…

The last time they tested the diesel generator failover at a customer's site, the generator went on just fine, but then it did not want to switch to mains again. The whole building was powered by the generator for almost two days, until they managed to convince the generator to switch.

Re: How I Fired Myself

#337

Post by batalismo » Thu, Feb 28, 2013, 7:21 AM UTC

lol

Re: How I Fired Myself

#338

Post by jgeerts » Thu, Feb 28, 2013, 7:29 AM UTC

This article sounds so incredible to me, I think I might have been holding my breath reading it. These are two major mistakes that the company is responsible for, not the author. Why would they let anyone in on the production password and do direct queries onto that database instead of working on a different environment, it's laughable that they sent this to their customers admitting their amateurism. Secondly, no backups? At my previous project, a similar thing happened to our scrum master, he accidently dropped the whole production database in some kind of the same situation. The database was back up in less than 10 minutes with an earlier version. It's still a mistake that should not be possible to make, but when it happens you should have a backup.

Re: How I Fired Myself

#339

Post by Kesty » Thu, Feb 28, 2013, 7:41 AM UTC

konstruktor wrote: post #167

Kesty wrote: post #76

Earlier quoted context omitted.

Everyone makes, has made and will make mistakes. Junior/Senior is not important. You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer. You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.

Risk avoidance (decent staging) and risk mitigation (backups) are two mostly orthogonal aspects of risk management. Often, a backup will be a good first step for a totally messed up system. However, saying that mistakes will aways reach the outer layer to discount the value of risk avoidance is talking about the possibility of risk realisation where what matters is probability.

I'm not trying to discount the value of risk avoidance they are both important and should both be used, but mitigation should always be the priority of the two.

1) When you have neither you should focus on risk mitigation first. 2) Having a great and complex risk avoidance policy in place is a good thing but doesn't mean that you need a lesser mitigation system.

Re: How I Fired Myself

#340

Post by ptaipale » Thu, Feb 28, 2013, 7:43 AM UTC

trotsky wrote: post #324

potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

When you think about it it's almost a logical certainty that he'd take the fall. Any company collectively able to understand that the actual failure was inadequate safeguards would have been able to see it coming and presumably would have prevented it from happening. If you're so inexperienced that you expect no one will ever make a mistake you'll obviously assume that the only problem was that someone made one. It's…

Indeed. The news here is not that a junior developer made an error with SQL db (everyone makes them, eventually), nor that the company did not have proper safeguards against problems (this happens far too often). The news is that despite such basic management incompetence, the company had been able to get a large number of paying customers.