How I Fired Myself • Hacker News Board

Re: How I Fired Myself

#341

Post by ftay » Thu, Feb 28, 2013, 7:51 AM UTC

cybernomad99 wrote: post #313

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

To the credit of the management, they did not fire him. He resigned. But the coworkers felt he was responsible personally. That makes a uneasy work environment.

That's not much to give credit for. They could certainly have done more to help him recover from this.

Re: How I Fired Myself

#342

Post by quanticle » Thu, Feb 28, 2013, 8:14 AM UTC

pnathan wrote: post #173

x3ord wrote: post #153

Earlier quoted context omitted.

Exactly. I'm ashamed that my first reaction reading this was to blame OP. But in the 2 min it took to read the post I had come full circle to wondering what kind of terribly run company would allow this to happen--I guess the type that hires philosophy majors straight out of college without vetting their engineering skills.

> the type that hires philosophy majors straight out of college without vetting their engineering skills. Or hires them and then adequately educating them in software development.

Or one that doesn't have a process to ensure that backup and recovery procedures when something like this happens are as painless as possible.

Re: How I Fired Myself

#343

Post by bart42_0 » Thu, Feb 28, 2013, 8:34 AM UTC

You can't make an omelette without breaking eggs.

Clicking on 'delete' with the user table selected was not very wise. The software maybe even asked 'Are you sure?' and of course you reply 'yes'.

But operating your company with proper recovery tools is a bit like climbing Mount everest without a rope.

If something goes wrong you are in deep sh.t.

Re: How I Fired Myself

#344

Post by kunil » Thu, Feb 28, 2013, 9:01 AM UTC

But why did you cleared users table at the first place? I don't get it.

Re: How I Fired Myself

#345

Post by flayman » Thu, Feb 28, 2013, 9:19 AM UTC

This is not your fault. Not really. And it's galling that the company blamed the incident on the workings of a 'junior engineer'. There was NO DATABASE BACKUP! For Christ's sake. This is live commercial production data. No disaster recovery plan at all. Zilch. And to make matters worse, you were expected to work with a production database when doing development work. This company has not done nearly enough to mitigate serious risks. I don't blame you for quitting. I would. I hope you have found or manage to find a good replacement role.

Re: How I Fired Myself

#346

Post by arethuza » Thu, Feb 28, 2013, 9:23 AM UTC

reinhardt wrote: post #241

happycube wrote: post #225

Earlier quoted context omitted.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

Sounds completely possible to me - I have worked in a lot of environments and at some places I've seen decisions that make cancelling backups look like an act of genius.

One interesting observation I can make: no correlation between excellence in operations and commercial success!

Re: How I Fired Myself

#347

Post by eloisant » Thu, Feb 28, 2013, 9:28 AM UTC

mbell wrote: post #168

eloisant wrote: post #25

Earlier quoted context omitted.

Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected. What's more questionable is: * Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter. * Junior developers don't need an access to production machine, only sysops and maybe the technical PM. * No bac…

I'll add another one: * No Foreign Keys Attempting to clear the table should have just thrown a constraint violation error.

Well, depending how you configure your cascade clearing the user table could have cleared all the other tables also :)

"on delete cascade"!

Re: How I Fired Myself

#348

Post by ptaipale » Thu, Feb 28, 2013, 9:40 AM UTC

jakejake wrote: post #159

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

My hypothesis is that it's a game company and all of the focus was on the game code. The lowly job of maintaining the state server was punted off to the "junior dev" just out of school. Nobody was paying attention. It was something that just ran. They paid the price of ignoring what was actually the most critical part of their business.

I disagree slightly. If you're a game company, your most critical part of the business is the game.

Even if you have a rock-solid database management, backup, auditing etc process, if your game is not playable, you won't have any data that you could lose by having a DB admin mis-click.

Still, not handling your next-most-critical data properly is monumentally stupid and a collective failure of everyone who should have known.

Re: How I Fired Myself

#349

Post by Nick_C » Thu, Feb 28, 2013, 9:48 AM UTC

blowski wrote: post #62

caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake. When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts: 1. Doctors do make mistakes. Lots of them. All the time. 2. Ev…

The aviation industry is the same. Most aviation authorities around the world have a no-fault reporting system so that fixes can get implemented without pilots worrying about losing their job.

Re: How I Fired Myself

#350

Post by ptaipale » Thu, Feb 28, 2013, 9:58 AM UTC

juan_juarez wrote: post #129

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.

Not really. You obviously haven't (yet) done anything like update t1 set status=0; where status=4; when you wanted to release (set status to 0) objects that are stuck in state 4, and let all other objects keep their existing statuses.

This is an easy mistake to make on command line. I hate GUIs too but not having one doesn't really help when your fundamental operating model is wrong.