31–40 of 424 posts
Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.
No, the CEO was at fault, as was whoever let you develop against the production database.
If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.
I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.
Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.
And they still have an empty users table.
http://krugman.blogs.nytimes.com/2011/08/11/academia-nuts/
What a screw up!
These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.
[†] They've long been removed.
http://dealbook.nytimes.com/2012/08/03/trading-program-ran-a...
1) Not having backups is an excuse-less monumental fuckup.
2) Giving anyone delete access to your production db, especially a junior dev through a GUI tool, is an excuse-less monumental fuckup.
Hopefully they rectified these two problems and are now a stronger company for it.
1. Work directly on the production database 2. Not have daily backups 3. Not have data migrations in place for these kinds of situations
needs to be fired immediately. My guess is it was one of the 'senior' engineers and that the author only worked with what they gave him.
I've worked with all kinds of bozos but I've never seen this kind of incompetence. Ridiculous.
Earlier quoted context omitted.
More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected. What's more questionable is: * Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter. * Junior developers don't need an access to production machine, only sysops and maybe the technical PM. * No bac…
Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.
We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs. My experience is that this is the norm, not the exception.
You did them more good than harm. 1) Not having backups is an excuse-less monumental fuckup. 2) Giving anyone delete access to your production db, especially a junior dev through a GUI tool, is an excuse-less monumental fuckup. Hopefully they rectified these two problems and are now a stronger company for it.