Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

171–180 of 424 posts

Re: How I Fired Myself

#171

Post by jcromartie »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Indeed. If it had been a sporadic hardware failure, they would have been exactly as screwed. The fact that they gave an overworked junior dev direct read/write access to the production database is astounding.

Re: How I Fired Myself

#172

Post by danso »

paganel wrote: post #140
ritchiea wrote: post #106

Earlier quoted context omitted.

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

Heh. Another way I would say that is that "discretion is the better part of valor" And, to rip-off Hitchhiker's Guide to the Galaxy, "cowardice is the better part of discretion"...in that if fear makes you judiciously check your backups and write tests, then that's not a bad thing at all.

Re: How I Fired Myself

#173

Post by pnathan »

x3ord wrote: post #153
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Exactly. I'm ashamed that my first reaction reading this was to blame OP. But in the 2 min it took to read the post I had come full circle to wondering what kind of terribly run company would allow this to happen--I guess the type that hires philosophy majors straight out of college without vetting their engineering skills.

> the type that hires philosophy majors straight out of college without vetting their engineering skills.

Or hires them and then adequately educating them in software development.

Re: How I Fired Myself

#174

Post by doktrin »

I found myself doing very much this my very first day on the job working for a software startup.

We had a Grails app that acted as a front end for a number of common DB interactions, which were selected via a drop down. One of these (in fact, the default) action was titled "init DB". Of course, this would drop any existing database and initialize a new one.

When running through the operational workflow with our COO on the largest production database we had, I found myself sleepily clicking through the menu options without changing the default value. I vividly remember the out of body experience the OP describes, and in fact offered to fire myself on the spot shortly thereafter.

It's fun to laugh about in hindsight, but utterly terrifying in the moment - to say nothing of the highly destructive impact it had on my self confidence.

Re: How I Fired Myself

#175

Post by oseibonsu »

potatolicious wrote: post #124
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I believe the company may be http://www.klicknation.com/games/ Can anyone confirm?

Re: How I Fired Myself

#176

Post by gte910h »

He worked at a company stupid enough to test on the prod databases without tools to safely clear them. The former is stupid, the later is REALLY stupid.

This is a multi-layer failure and almost none of the blame falls on him. Stupid compounded stupid, and this guy did nothing more than trip over the server cord several people who knew better stupidly ran past his cube exit.

Re: How I Fired Myself

#177

Post by johngalt »

"How I was setup to fail."

Who asks junior engineer to develop directly on live systems with write access and no backup? Are you kidding me?

Edit: No one ever builds a business thinking about this stuff, until something like this happens. There are people who have learned about operations practices the hard way, and those who are about to. They hung the author out to dry for a collective failure and it shows that this shop is going to be taught another expensive lesson.

Re: How I Fired Myself

#178

Post by zdgman »

novax81 wrote: post #166
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

The difference between working with bad bosses and good bosses really shows itself when there's a disaster going on. A mere few months into my current job, I ran an SQL query that updated some rows without properly checking one of the subqueries. Long story short - I fucked up an entire attribute table on a production machine (client insisted they didn't need a dev clone). I literally just broke down and started pani…

I really wish this was at the top. Everyone will fuck up at some point (even your best engineer). Whether you learn anything from fucking up really determines how bad your mistake was.

Re: How I Fired Myself

#179

Post by gte910h »

jcromartie wrote: post #171
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Indeed. If it had been a sporadic hardware failure, they would have been exactly as screwed. The fact that they gave an overworked junior dev direct read/write access to the production database is astounding.

It's not that they gave him r/w at all that's so criminally stupid. It's that the required him to clear a table manually, using generic full access tools, over and over.

In reality, this should have been re-factored to the dev db.

If it couldn't be, the junior dev should have been given access to the raids table alone for writes.

Lastly the developer who didn't back up this table is the MOST to blame. Money was paid for the state in that table. That means people TRUST you to keep it safe.

I count tons of people to blame here. I don't really see the junior dev as one of them.

Re: How I Fired Myself

#180

Post by kibwen »

"I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago."

Here's something I don't get: didn't Rackspace have their own daily backups of the production server, e.g. in case their primary facility was annihilated by a meteor (or some more mundane reason, like hard drive corruption)?

Regardless, here's a thought experiment: suppose that Rackspace did keep daily backups of every MySQL instance in their care, even if you're not paying for the backup service. Now suppose they get a frantic call from a client who's not paying for backups, asking if they have any. How much of a ridiculous markup would Rackspace need to charge to give the client access this unpaid-for backup, in order to make the back-up-every-database policy profitable? I'm guessing this depends on 1) the frequency of frantic phone calls, 2) the average size of a database that they aren't being paid to back up, and 3) the importance and irreplacebility of the data that they're handling (and 4) the irresponsibility of their major clients).

Post reply on HN