Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

71–80 of 424 posts

Re: How I Fired Myself

#71

Post by mootothemax »

danielh wrote: post #63
mootothemax wrote: post #57

If you ever notice that your employer or client isn't backing up important data, take a tip from me: do a backup, today, in your free time, and if possible, and again in your free time, create the most basic regular backup system you can. When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

Excellent point! Any tips on how to avoid that, other than not taking the data home / copying to personal Dropbox-type things?

Re: How I Fired Myself

#73

Post by elomarns »

Although the author of the post obviously did a huge mistake, he is far from being the actual responsible for the problem that follows his mistake. It's the job of the CTO to make sure no one can harm the company main product this way, accidentaly or not.

He could never write code against the production database when developing new features. And if he was doing it, it wasn't his fault, considering he was a junior developer.

And who the hell is so stupid to don't have any recent backup for the database used by a piece of software that provides millions of revenue?

In the end, when you do such a shity job protecting your main product, shit will eventually happens. The author of the post was merely a agent of destiny.

Re: How I Fired Myself

#74

Post by j_baker »

mootothemax wrote: post #32

The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue". No, the CEO was at fault, as was whoever let you develop against the production database. If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again . Taking things further, they could have asked you to find other wor…

To be fair, if the CEO were willing to take those steps, the company would probably not have a deleted USERS table.

Re: How I Fired Myself

#75

Post by ry0ohki »

I think all developers have to do something like this at some point to get the compulsion I have which is backups to the extreme. I can never have enough backups. Any DROP/ALTER type change I make a backup. (And also learned to pretty much never work on a production db directly, and in the event I need to, doing a complete test of a script in staging first...)

Re: How I Fired Myself

#76

Post by Kesty »

mkrecny wrote: post #54
lallouz wrote: post #31

I would love to see some reflection on this story from OP. What do you think you learned from this experience? Do you think your response was appropriate? What would you have done differently? Are you forever afraid of Prod env? Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.

OP here. I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.

Everyone makes, has made and will make mistakes. Junior/Senior is not important.

You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer.

You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.

Re: How I Fired Myself

#77

Post by jiggy2011 »

blowski wrote: post #62
caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake. When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts: 1. Doctors do make mistakes. Lots of them. All the time. 2. Ev…

> If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.

This is absolutely it, of course it is possible to become so risk averse that you never actually succeed in getting anything done and there are certainly organisations that suffer from that (usually larger ones).

However some people seem to take the view that it is impossible to protect oneself from all risks therefor it is pointless protecting from any of them.

The good news is that usually protecting against risks tends to get exponentially more expensive as you add "nines" therefor having a 99% guarantee against data loss is a lot cheaper than a 99.999% guarantee.

Having a cronjob that does a mysqldump of an entire database, emails some administrator and then does rsync to some other location (even just a dropbox folder) is something that is probably only a couple of hours work.

Re: How I Fired Myself

#78

Post by danielweber »

I feel tremendous empathy for this guy.

Not because I've done this. But because there but for the grace of God go I. It wouldn't take much changing in the universe for me to be this guy.

I'm very glad he's posting it, and I hope everyone reads it, so you can learn from his very painful mistake instead of committing it yourself.

Re: How I Fired Myself

#79

Post by MBCook »

xentronium wrote: post #16

This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible. Anecdote: I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important lat…

Ug. MyISAM.

More than once (over the last few years) I've been doing some important update. I tend to do it the same way.

  START TRANSACTION;
  --run SQL--
  --check results--
  COMMIT; or ABORT TRANSACTION;
Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup.

So you always have to make a backup and check that the tables are defined they way they should be. Nothing is quite as much fun as the first time you delete a bunch of critical data off a production system. Luckily the table was small, basically static, and we had backups so it was only a problem for ~5 minutes.

Re: How I Fired Myself

#80

Post by MBCook »

petriw wrote: post #70
arethuza wrote: post #49

Earlier quoted context omitted.

Out of interest, why not create a database user account that is read only and use that?

Just remember to always verify it's still read only. Or a coworker will find the login in your scripts, repurpose it, then notice they need more rights and "fix" the account for you.

Plus read-only isn't a guarantee. You can't write data, but you can run a bad select or join that ends up effectively locking the database.

SELECT * FROM my_200_GB_table will always be there.

Post reply on HN