Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

211–220 of 424 posts

Re: How I Fired Myself

#211

Post by ajross »

indrax wrote: post #184
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind.

Not once have I seen a full backup restore tested. Not once have I seen a network failure simulated (though I've seen several system failures due to "kicking out a cable" that sort of acts as a proxy for that technique). On multiple occasions I've seen systems taken down by single points of failure[1] that weren't forseen, but probably could have been.

[1] My favorite: the whole closet went down once because everything was plugged into a single, very expensive, giant UPS that went poof. $40/system for consumer batteries from Office Depot would have been a much better bet. And the best part? Once the customer service engineer replaced whatever doodad failed and brought the thing back up? They plugged everything right back into it.

Re: How I Fired Myself

#212

Post by doktrin »

roryokane wrote: post #188
doktrin wrote: post #174

I found myself doing very much this my very first day on the job working for a software startup. We had a Grails app that acted as a front end for a number of common DB interactions, which were selected via a drop down. One of these (in fact, the default) action was titled "init DB". Of course, this would drop any existing database and initialize a new one. When running through the operational workflow with our COO o…

How did the company deal with the loss of that database? Did they actually have backups, and just restored the data? Did they reconstruct the data from other sources?

In our case we had periodic backups, and together with filesystem logs were able to restore most of the data. However, we were hosting highly sensitive data and the work being done was time critical. The downtime was therefore not popular with our clients, who were losing ~$15k per hour offline.

Re: How I Fired Myself

#213

Post by j_baker »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

"Nobody cares about technical infrastructure. Our customers don't pay us for engineering rigor. We need to just ship!"

Of course the person saying that is likely to care about technical infrastructure when it costs them money and/or customers due to being hacked-together.

Re: How I Fired Myself

#214

Post by j_baker »

potatolicious wrote: post #124
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual

You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

Re: How I Fired Myself

#215

Post by chris_mahan »

j_baker wrote: post #213
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

"Nobody cares about technical infrastructure. Our customers don't pay us for engineering rigor. We need to just ship!" Of course the person saying that is likely to care about technical infrastructure when it costs them money and/or customers due to being hacked-together.

Nobody cares about the diameter of the cylinders in the engine, all they care about is going from point A to point B safely, affordably, in style.

Re: How I Fired Myself

#216

Post by praptak »

They should reward him. Seriously, anyone who exposed such a huge weakness deserves a reward. He limited the damage to only 10k users' data loss. With such abysmally crappy practices the damage would happen anyway only perhaps with 30k users and who knows what else instead of a mere 10k.

Re: How I Fired Myself

#217

Post by Plutor »

indrax wrote: post #184
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

Minor correction: the Chaos Monkey was Netflix's innovation. It just happened to be implemented on Amazon's cloud. It would have been just as useful if they had their own colocated servers or used a different cloud computing provider.

Re: How I Fired Myself

#218

Post by r00fus »

indrax wrote: post #184
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

Apple did this before Amazon or Netflix in this regard [1], but the point needs to be made that a system needs to be tested and not just in a controlled aseptic way, because the real world isn't.

Another story supporting Chaos Monkey is what the Obama team did for their Narwhal infrastructure - they staged outages and random failures to prepare for their big day, meanwhile Romney's team who outspent the Obama team at least an order of magnitude, had their system fail on e-day.

[1] http://folklore.org/StoryView.py?story=Monkey_Lives.txt

Re: How I Fired Myself

#219

Post by alexkus »

ajross wrote: post #211
indrax wrote: post #184

Earlier quoted context omitted.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

> Not once have I seen a network failure simulated.

Reminds me of the webserver UPS setup at a previous company.

The router (for the incoming T1) and the webserver were plugged in to the UPS.

UPS connected (via serial port) to webserver. Stuff running on webserver to poll whether UPS running from mains power or batteries and send panic emails if on batteries (for more than 60 seconds) and eventually shutdown the webserver cleanly if UPS power dropped below 25%.

Thing not plugged in to UPS: DMZ Network switch (that provided the connectivity between webserver and router).

Re: How I Fired Myself

#220

Post by hga »

ajross wrote: post #211
indrax wrote: post #184

Earlier quoted context omitted.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Bingo. As I like to emphasize, people don't care about backups---this company certainly didn't---they care about restores.

And almost no one is willing to put up the money to do compete testing of restore paths, let along statistically making sure they continue to work.

Post reply on HN