Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

221–230 of 424 posts

Re: How I Fired Myself

#221

Post by pbhjpbhj »

kibwen wrote: post #180

"I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago." Here's something I don't get: didn't Rackspace have their own daily backups of the production server, e.g. in case their primary facility was annihilated by a meteor (or some more mundane reason, like hard drive corruption)?…

Nope, not going to happen. At least one good reason and that is that if Rackspace leak your data via a backup they're going down to the tune of millions.

Yes it would be nice if Rackspace could speculatively create a backup but they'd be dancing on ice doing so.

Re: How I Fired Myself

#222

Post by pc86 »

r00fus wrote: post #218
indrax wrote: post #184

Earlier quoted context omitted.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

Apple did this before Amazon or Netflix in this regard [1], but the point needs to be made that a system needs to be tested and not just in a controlled aseptic way, because the real world isn't. Another story supporting Chaos Monkey is what the Obama team did for their Narwhal infrastructure - they staged outages and random failures to prepare for their big day, meanwhile Romney's team who outspent the Obama team at…

I'd like to see a source for Romney outspending the Obama team "at least" 10x, because while I can speak from experience that ORCA was a gigantic piece of shit, it's not like the Obama people were struggling to pay their bills.

Re: How I Fired Myself

#223

Post by downandout »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have been fired.

I've had more than my fair share of failures in the start up world. It always drives me crazy to see internet companies that have absolutely no technical ability succeed, while competing services that are far superior technically never get any traction.

Re: How I Fired Myself

#224

Post by themonk »

krallja wrote: post #142
juan_juarez wrote: post #129

Earlier quoted context omitted.

Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.

Neither of those things would have fixed this problem. 1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids" 2. Transaction complete. ... ... "oops!"

Why type "TRUNCATE TABLE Raids" every time? Type it once, test it, save as script, schedule that script.

Re: How I Fired Myself

#225

Post by happycube »

downandout wrote: post #223
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have…

And worse, they had a backup service, and then dropped it to save money.

Re: How I Fired Myself

#226

Post by SonicSoul »

this is insanity! it was already pointed out in comments but i still can't believe a company that mature (actually having 1000's of users and millions in revenue!!!) would omit such a basic security precaution. Giving [junior?!] developers free reign in production database and no backups???? seriously, the CTO should have been fired on the spot instead of putting blame on developer.

no matter how careful you are (i'm extremely careful) when working with data, if you're working in dev/qa/uat/prd sooner or later someone on dev team will execute on wrong environment.

Re: How I Fired Myself

#227

Post by pmelendez »

tlrobinson wrote: post #134
pmelendez wrote: post #23

There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident. 1) A cron job for the manually task you were doing. 2) Not working directly on production. 3) Having daily backups And this could happened to anybody. After midnight any of us are at junior level and very prone to do this…

Hopefully prioritized in reverse order

Touché! I didn't intended to prioritize my list but you are right :)

Re: How I Fired Myself

#228

Post by bashinator »

caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

Or d) too scared/timid to be an effective employee and learn new skills.

Re: How I Fired Myself

#229

Post by elicash »

I think it's admirable that you stayed long enough to help fix everything before quitting, despite it being rough -- even though, as others have said, others screwed up even bigger than you did.

Re: How I Fired Myself

#230

Post by lexilewtan »

So many structures in life are based around 'not fucking up.' We protect our assets & our dignity as if they mean anything; and yet at the end of the day nobody knows what the fuck is going on.

really simple, revealing story. kudos.

Post reply on HN