Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

41–50 of 424 posts

Re: How I Fired Myself

#41

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."

I asked similar, and agree that it's a really useful question.

I think it's an especially great one for startups, as successful candidates are more likely to come into contact with production systems.

For these positions you not only want people capable of recovering accidents, but also people who have screwed up because, conversely, they've been trusted not to screw up. Those who've never been trusted enough to not damage a system are unlikely to be of much use.

Re: How I Fired Myself

#42
If your senior management/devs are worth anything, they were already aware that this was a possibility. There is no excuse for what ostensibly appears to be a total lack of a fully functioning development & staging environment--not to mention any semblance of a disaster recovery plan.

My feeling is that whatever post-incident anger you got from them was a manifestation of the stress that comes from actively taking money from customers with full knowledge that Armageddon was a few keystrokes away. You were just Shaggy pulling-off their monster mask at the end of that week's episode of Scooby Doo.

Re: How I Fired Myself

#43
post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

[deleted]

Re: How I Fired Myself

#44
It's unlikely that the database had no foreign keys related to users table. And if so DBMS should have prevented deleting all users from the table.

Perhaps the Database Designer also failed his job. As well as the guys who cancelled backups and set up dev environment.

Re: How I Fired Myself

#45
I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower.

In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulating those.. reading over my scripts for hours.. dumping table backups before executing any scripts.. not executing scripts in the middle/end of the day, only mornings when I was fresh etc etc.

I didn't mess up, but I remember how incredibly nerve wracking that was, and I can relate to the massive amount of responsibility it places on a "junior" programmer. It just should never be done. Like others have said, you should never have been in that position. Yes, it was your fault, but this kind of responsibility should never have been placed on you (or anyone, really). Backing up all critical data (what kind of company doesn't backup its users table?! What if there had been hard disk corruption?), and being able to restore in minimum time should have been dealt with by someone above your pay grade.

Re: How I Fired Myself

#46
post #31

I would love to see some reflection on this story from OP. What do you think you learned from this experience? Do you think your response was appropriate? What would you have done differently? Are you forever afraid of Prod env? Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.

[deleted]

Re: How I Fired Myself

#47
Mistakes happen, and there should have been better safeguards -- backups, locking down production, management oversight.

But, I actually applaud how he tried to take responsibility for his actions and apologized. Both "junior" AND "senior" people have a hard time doing this. I've seen experienced people shrug and unapologetically go home at 6pm after doing something equivalent to this.

The unfortunate thing here seems to be that he took his own actions so personally. He made an honest mistake, and certainly there were devastating consequences, but it's important to separate the behavior from the person. I hope he realizes this in time and forgives himself.

Re: How I Fired Myself

#49

I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower. In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulat…

Out of interest, why not create a database user account that is read only and use that?

Re: How I Fired Myself

#50
I've did a very similar thing after one year working at my company instead of clearing the whole user table I replaced every user information with my account information.

I forgot to copy the WHERE part of the query .....

The only difference is that it was policy to manually do a backup before doing anything on production and the problem was restored in less than 10 minutes. Even if I had forgotten to make a backup manually we had a daily complete backup and an incremental one every couple of hours.

Post reply on HN