This is handing a heart for transplant to the Post Office and hoping for the best.
How I Fired Myself
141–150 of 424 posts
Re: How I Fired Myself
#142Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…
Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.
Neither of those things would have fixed this problem.
1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids"
2. Transaction complete. ... ... "oops!"
Re: How I Fired Myself
#143Earlier quoted context omitted.
Exactly what you said. Rigor is truly the right word to use here. Cancelling your db backups is basically asking for a disaster. I'm not sure i have ever been at job that didnt require a db backup for some reason at some time.
To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?
The junior dev was 22 at the time and fresh out of school.
I'm assuming the senior devs were 23 and had a year on the job. The principal devs were a ripe 24, otherwise known as old men. :-)
Re: How I Fired Myself
#144News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…
>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…
I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it.
Does Amazon Glacier offer this?
Re: How I Fired Myself
#145More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.
More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…
Exactly this.
Re: How I Fired Myself
#146[deleted]
Re: How I Fired Myself
#147I really hope this is some kind of hoax and no real company was operating like that in 2010.
Re: How I Fired Myself
#148This happened to me once on a much smaller scale. Forgot the "where" clause on a DELETE statement. My screwup, obviously.
We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use."
(Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah! We're backing up! Yeah! Well, some things. Most things. The files that don't get like, used and stuff.")
That day was one of the lowest feelings I ever had, and that screwup "only" cost us a few thousand dollars as opposed to the millions of dollars the blog post author's mistake cost the company. I literally can't imagine how he felt.
Re: How I Fired Myself
#149Earlier quoted context omitted.
To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?
The author makes mention of using a UI to connect to their db. If i was in a position over there i can see myself writing a script to clear out the tables i wanted. This reduces errors, but not the risk.
And yet, the net result is the same; right-click, clear table, or ./clearTable.sh. Both are human actions, and both are fallible. What if some prankster edited clearTable.sh to do the users table instead of the raids table? What if he did it himself to test something?
Re: How I Fired Myself
#150What company that makes millions in revenue doesn't replicate their database or at least have snapshots?
What engineer uses any GUI to administrate MySQL?
This story feels totally unreal to me (unreal as in just crazy, not disbelief).