the monumental fuck up was cancelling mysql backup and having all engineers work directly with the production database, what you did was INEVITABLE..
How I Fired Myself
161–170 of 424 posts
Re: How I Fired Myself
#162News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory:
One day Intel's yields suddenly went to hell (that's the ratio of working die on a wafer to non-working, and is a key to profitability). And no matter how hard they tried, they could only narrow it down to the wafers being contaminated, but the wafer supplier swore up and down they were shipping good stuff, and they were. So eventually they tasked a guy to follow packages from the supplier all the way to the fab lines, and he found the problem in Intel's receiving department. Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....
His point is that you can have a Fortune 500 company, normally thought to be stable companies that won't go "poof" without ample warning, in which there are many more people than in previous kinds of companies who can very quickly kill it dead.
Re: How I Fired Myself
#163News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…
This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…
slowly and work in fear businesses usually collapse. the technical problem eventually becomes a business problem and no one realizes it can be solved at the source level.
it first builds inefficieny at the technical level then at the business level, finally it causes issues at the cultural level, and thats when the smart people start leaving.
Re: How I Fired Myself
#164My name is Myles. I read this and felt like I was looking into a crystal ball. Fortunately, my work doesn't require I interact with the production database (yet). Gulp.
Re: How I Fired Myself
#165Earlier quoted context omitted.
We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs. My experience is that this is the norm, not the exception.
As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.
Well you should certainly look for a team that gives you a good impression with their dev process.
But, things are not always perfect, even on great teams. Not saying its normal to destroy your production database! But even in good shops it's a constant challenge to stay organized and do great work. Look for a team that is at least trying to do great work, rather than a complacent team.
Re: How I Fired Myself
#166News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…
The difference between working with bad bosses and good bosses really shows itself when there's a disaster going on.
A mere few months into my current job, I ran an SQL query that updated some rows without properly checking one of the subqueries. Long story short - I fucked up an entire attribute table on a production machine (client insisted they didn't need a dev clone). I literally just broke down and started panicking, sure that I'd just lost my new job and wouldn't even have a reference to run on for a new one.
After a few minutes of me freaking out, my boss just says: "You fucked up. I've fucked up before, probably worse than this. Let's go fix it." And that was it. We fixed it (made it even better than it was, as we fixed the issue I was working on in the process), and instead of feeling like an outcast who didn't belong, I learned something and became a better dev for it.
Re: How I Fired Myself
#167Earlier quoted context omitted.
OP here. I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.
Everyone makes, has made and will make mistakes. Junior/Senior is not important. You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer. You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.
Risk avoidance (decent staging) and risk mitigation (backups) are two mostly orthogonal aspects of risk management. Often, a backup will be a good first step for a totally messed up system.
However, saying that mistakes will aways reach the outer layer to discount the value of risk avoidance is talking about the possibility of risk realisation where what matters is probability.
Re: How I Fired Myself
#168Earlier quoted context omitted.
More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected. What's more questionable is: * Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter. * Junior developers don't need an access to production machine, only sysops and maybe the technical PM. * No bac…
I'll add another one:
* No Foreign Keys
Attempting to clear the table should have just thrown a constraint violation error.
Re: How I Fired Myself
#169It's unlikely that the database had no foreign keys related to users table. And if so DBMS should have prevented deleting all users from the table. Perhaps the Database Designer also failed his job. As well as the guys who cancelled backups and set up dev environment.
Well, the foreign key could have been set up to cascade deletes, in which case they would have been extra-screwed.
Re: How I Fired Myself
#170Good lord, that's unbelievable! If millions of dollars are riding on a database, they should have spent a few thousand to replicate the database, run daily backups and maintain large enough rollback buffers to reverse an accidental DROP or DELETE.
We've all screwed up at various times (sometimes well beyond junior phase), but not to have backups.... That's the senior management's fault.