Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

61–70 of 424 posts

Re: How I Fired Myself

#62

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake.

When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts:

1. Doctors do make mistakes. Lots of them. All the time.

2. Even an average doctor is paid an awful lot more than me.

3. Doctors have other people analysing where things can go wrong, and recommending fixes.

If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.

Re: How I Fired Myself

#63

If you ever notice that your employer or client isn't backing up important data, take a tip from me: do a backup, today, in your free time, and if possible, and again in your free time, create the most basic regular backup system you can. When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

Re: How I Fired Myself

#65
Your CEO was correct. He should have also said the same thing to the guy who cancelled backups as well...and the guy who never put in place and periodically tested a disaster recovery plan. So much fail in this story, but mistakes happen and I've had my share as well.

I once (nah twice) left a piece of credit card processing code in "dev mode" and it wasn't caught until a day later costing the company over 60k initially. Though they were able to recover some of the money getting the loss down to 20k. Sheesh.

Re: How I Fired Myself

#66
post #7
post #2

Whoever cancelled the backups was equally responsible

More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.

Well - to be fair - if the company's practices include developing on the production database and not doing daily/weekly backups, then hiring an inexperienced Philosophy major is the least of their problems.

Also, someone who actually had the development experience and knowledge of better-practice would not have taken that position.

Re: How I Fired Myself

#67
Stuff like this happens. The best thing to prevent something like this is to completely sever the line between production and development. I've worked with companies that work directly on the production database. It's horrible. How can the person in charge of managing the workflow expect something like this not to happen eventually?

Re: How I Fired Myself

#68
"The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals."

Holy crap. I know that _exact_ same feeling. I had to laugh. I know that out-of-body feeling all too well.

Re: How I Fired Myself

#69

I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower. In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulat…

Out of interest, why not create a database user account that is read only and use that?

It was a bunch of different tasks. For some, we did use a read only account. Other tasks (updating top 10 scores, updating the users table with their geo-ip based location etc) required write access.

Just to add some more thoughts based on other comments.. yes a lot of companies do stuff like this, particularly startups. The upside in these situations is that you end up learning things extremely quickly which wouldn't be possible in a more controlled environment. However not having backup and restore working is just ridiculous and I keep shaking my head at how they blamed the OP for this mistake. Unbelievable.

Re: How I Fired Myself

#70

I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower. In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulat…

Out of interest, why not create a database user account that is read only and use that?

Just remember to always verify it's still read only.

Or a coworker will find the login in your scripts, repurpose it, then notice they need more rights and "fix" the account for you.

Post reply on HN