Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

191–200 of 424 posts

Re: How I Fired Myself

#191

Post by grey-area »

jakejake wrote: post #159
grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

My hypothesis is that it's a game company and all of the focus was on the game code. The lowly job of maintaining the state server was punted off to the "junior dev" just out of school. Nobody was paying attention. It was something that just ran. They paid the price of ignoring what was actually the most critical part of their business.

Sounds convincing. I guess now they'll realise their mistake though as the servers are critical to their business.

Re: How I Fired Myself

#192

Post by damm »

potatolicious wrote: post #124
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

When things like this happen, you have to realize there is more than one 'truth'.

There is the truth that here is someone who truncated the users table and because of that it caused the company great harm.

Here's another 'truth'.

1. The company lacked backups 2. The junior developer was on a production database.

Note: I'm from the oldschool of sysops who feel that you don't give every employee the keys to your kingdom. Not every employee needs to access every server, nor do they need the passwords for such.

3. Was there a process change? I doubt it, likely they made the employee feel like a failure every day and remind him of how he can be fired at any moment. So he did the only thing he could do to escape: Quit!

Horrible and wrong, if there was a good ethics lawyer around he would say it smells ripe of a lawsuit.

... That said, that lawyer and lawsuit won't fix anything.

Re: How I Fired Myself

#193

Post by jtchang »

I'm with everyone else in this thread: you screwed up but in reality it is EXPECTED.

Do you know why I have backups? Because I'm not perfect and I know one day I will screw up and somehow drop the production database. Or mess up a migration. Or someone else will. This is stuff that happens ALL THE TIME.

Your CEO/CTO should have been fired instead. It is up to the leadership to ensure that proper safeguards are in place to avoid these difficult conversations.

Re: How I Fired Myself

#195

Post by nuclear_eclipse »

oseibonsu wrote: post #175
potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I believe the company may be http://www.klicknation.com/games/ Can anyone confirm?

From the author's page, clicking "PAST" shows the following:

I've worked as a Software Engineer at TechStarsNYC, Klicknation(acquired), Shelby and I'm currently working on Followgen

So, yes.

Re: How I Fired Myself

#196

Post by sethammons »

Piskvorrr wrote: post #121
islon wrote: post #18

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a…

That's why the very first package installed after ssh-server should be molly-guard. Saved my butt a few times...

molly-guard -- looks great. I've always aliased my commands so if I accidentally type 'po' while in an ssh session on a production server then it complains. On my local dev box, 'po' is just quicker for me to power off the box. It has saved me at least once.

Re: How I Fired Myself

#198

Post by rhizome »

The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue". His co-founder (remotely present via Skype) chimed in "you're lucky to still be here".

This is when you should have left. That's no way to manage a crisis.

Re: How I Fired Myself

#199

Post by jiggy2011 »

ismarc wrote: post #104
caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

Part of our culture/personality/team fit questions is like this. We also have one that's something like "tell me about a time you failed a commitment (deadline, etc) and how you handled it". Numerous people with over a decade claim perfect records, frequently blaming all those around them as having failed. It's been a real easy way to eliminate candidates, particularly because almost every team member we have gave an…

Interesting, though perhaps they misinterpreted the question. Some companies seem to have a culture where you never admit failure and perhaps they assumed that was the answer you were looking for?

Re: How I Fired Myself

#200

Post by ferrouswheel »

Ah, I remember being called away from my new year holiday when an engineer dropped our entire database.

This happened because they didn't realise they were connected to the production database (rather than their local dev instance). We were a business intelligence company, so that data was vital. Luckily we had a analysis cluster we could restore from, but afterwards I ensured that backups were happening... never again.

(Why were the backups not already set up? Because they were not trivial due to the size of the cluster and having only been CTO for a few months there was a long list of things that were urgently needed)

Post reply on HN