This has been stated by others, but it's not the author's fault. It's totally idiotic for a database like that not to have been regularly backed up. At worst, this should have been only a couple hours of down time while the database was restored.
How I Fired Myself
201–210 of 424 posts
Re: How I Fired Myself
#202Earlier quoted context omitted.
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…
I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...
We are all born naked, bloody, and screaming; the only thing we know is how to work a nipple. Everything else has to be learned.
One of Toyota's mantras is "If the student has failed to learn, the teacher has failed to teach." Their point is that managers are responsible for solving issues that come from employee ignorance, not line workers.
Re: How I Fired Myself
#203News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…
>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…
From a sysadmin:
Redundancy = Reduce the number of component failures that can lead to system failure (RAID, live replication, hot standby).
Backup = Recover from an obvious failure or overwite (Weekly full backups, daily differentials).
Archival = Recover from a non-obvious failure and/or malicious activity (WORM tapes, offsite backup).
As a failsafe against malicious sysadmins is to split up the responsibilities. The guy handling backups isn't handling archival etc...
Re: How I Fired Myself
#204Earlier quoted context omitted.
Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.
Neither of those things would have fixed this problem. 1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids" 2. Transaction complete. ... ... "oops!"
Well, to be fair it's harder to type User than it is to miss your rightclick by something that could have only been a few pixels.
Re: How I Fired Myself
#205Earlier quoted context omitted.
More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…
When things like this happen, you have to realize there is more than one 'truth'. There is the truth that here is someone who truncated the users table and because of that it caused the company great harm. Here's another 'truth'. 1. The company lacked backups 2. The junior developer was on a production database. Note: I'm from the oldschool of sysops who feel that you don't give every employee the keys to your kingdo…
The problem isn't who has the keys, it's how they're used. I don't care as much if a junior developer has the prod password; I care more about building an engineering and ops team that understands that dicking around with the prod database isn't okay. Sysops and DBAs are fallible too--I've seen a lot of old school shops that relied heavily on manual migration and configuration. Automate, test, isolate and expect failure!
Re: How I Fired Myself
#206This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible. Anecdote: I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important lat…
Reading those stories makes me realize how well thought-out the process at my work is: We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the stag…
When moving data from prod to other environments, consider a scrambler. E.g., replace all customer names with names generated from census data.
I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody.
Since your handle is perlgeek, you're already well equipped to do a streaming transformation of your SQL dump. :)
Re: How I Fired Myself
#207Earlier quoted context omitted.
As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…
This is the right thing to encourage but I just would like to add always have a backup. "Don't be afraid to break things as long as you have a backup". It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.
Use source control. You can always revert.
Re: How I Fired Myself
#208Earlier quoted context omitted.
To avoid mistakes like that is why I put the hostname and only the hostname plus one character in my shell prompt. (The other character is a # or $ depending on whether the user is root or not.)
At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.
Right on, I started doing this a few years ago. I really like it.
Re: How I Fired Myself
#209I cannot believe that people still don't have reliable back up in place.
My feeling is this: If you are in any way responsible for data that is not backed up, you should be fired or resign right now. You should never work in IT, in anyway, ever again. If you are the CEO of a company in a similar state, again, fire your self right now. Vow to never ever run a business again. This is 2013. And guess what? You still can't buy your unique data back from PCWorld. Your data is "the precious".
As for the treatment of this guy, IMHO, his employers were the worst kind of spineless cowards. This was 100% the fault of the management, and you know what? They know it. To not have backups is negligent, and should result in a high up firings. Yet these limp cowards sought to blame this kid. Pure corporate filth of the lowest order. Even the fact he was junior is irrelevant, any one could have done that, more likely a cocky senior taking some short cut. Let me tell you now, I have made a similar cock up, and I think I know it all. But I had backups, and lucky for me, it was out of business hours. Quick restore, and the users never knew. I did fess up to my team since I thought it had direct value as a cautionary tail.
Frankly, I am utterly amazed and gutted that such a thing can still happen. The corporate cowardice is sadly expected, but to not have backups is literally unforgivable negligence.
Yeah, Im quite fundamentalist about data and backups. I'd almost refer to my self as a backup jihadist.
Re: How I Fired Myself
#210Earlier quoted context omitted.
We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs. My experience is that this is the norm, not the exception.
As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.
I think it's entirely fair to haul out the Joel Test and say you'd just like to hear how they handle those things.
I honestly think the test is a little out of date, but if they say, "Well, instead of X we're doing better thing Y", that's a great answer.