Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

301–310 of 424 posts

Re: How I Fired Myself

#301

Post by jms »

ajross wrote: post #211
indrax wrote: post #184

Earlier quoted context omitted.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Back when I worked for a small ISP, we had a diesel generator in case the power went out longer than our UPS batteries would last. This provided a great sense of security until we decided to test the system by powering off the main break and... it didn't start!

It turns out the emergency stop button was pushed in. Easy enough for us to fix then, but if the power had gone out at 4am it would have been quite another matter.

After that incident, we turned off the main breaker to the building weekly. It was great fun, as most of our offices were in the same building. We had complaints for the first couple of months until everyone got used to it and had installed mini UPS's for their office equipment.

We did actually have to use the generator for real a while later. Someone had driven their car into the local power substation, and it was at least a month until it was fixed. Electricity was restored through re-routing fairly quickly, but until the substation was repaired we were getting a reduced voltage that caused the UPSs to slowly drain...

Re: How I Fired Myself

#303

Post by lkrubner »

Klicknation is hiring. Of themselves, they say:

"We make astonishingly fun, ferociously addictive games that run on social networks. ...KlickNation boasts a team of extremely smart, interesting people who have, between them, built several startups (successful and otherwise); written a novel; directed music videos; run game fan sites; illustrated for Marvel Comics and Dynamite Entertainment with franchises like Xmen, Punisher, and Red Sonja; worked on hit games like Tony Hawk and X-Men games; performed in rock bands; worked for independent and major record lables; attended universities like Harvard, Stanford, Dartmouth, UC Berkeley; received a PhD and other fancy degrees; and built a fully-functional MAME arcade machine."

And this is hilarious: their "careers" page gives me a 404:

http://www.klicknation.com/careers/

That link to "careers" is from this page:

http://www.klicknation.com/contact/

I am tempted to apply simply to be able to ask them about this. It would be interesting to hear if they have a different version of this story, if it is all true.

Re: How I Fired Myself

#304

Post by brittohalloran »

utunga wrote: post #274
ajross wrote: post #211

Earlier quoted context omitted.

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

I'll never forget when my boss was showing the girl scouts (literally) our very expensive UPS room. He explained how even if the power goes off we'll switch to batteries then switch over to generator power. See watch, he says - then flicked the switch. Fooomm... our entire office goes dark. This took down news information for a good chunk of Wellington finance for about half a day. (Fortunately Wellington, NZ is a ti…

Literally lol'd

Re: How I Fired Myself

#305

Post by tn13 »

I dont think this is author's fault. These kind of human mistakes are more than common. It is said that the top management actually assigned the blame to this young man. This was an engineering failure.

I can understand what this person must have gone through.

Re: How I Fired Myself

#306

Post by xradionut »

reinhardt wrote: post #241
happycube wrote: post #225

Earlier quoted context omitted.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

You would be amazed if I told you how many "vital" services are like the blog writers former employers: One button push from destruction.

Re: How I Fired Myself

#308

Post by joemoon »

dntrkv wrote: post #287
angersock wrote: post #261

Earlier quoted context omitted.

Having just helped a friend who was miffed at the idea of spending money on a new UPS for >$5k worth of networking equipment: do not be surprised. Penny-wise pound-foolish.

Spending $5k on a UPS is very different from not having backups of your production database which runs your multimillion dollar business. This story just doesn't add up.

You misunderstood the comment. Angersock is saying that the networking equipment cost more than $5k and the friend was unwilling to buy a UPS to protect the equipment.

Re: How I Fired Myself

#309

Post by artursapek »

xradionut wrote: post #306
reinhardt wrote: post #241

Earlier quoted context omitted.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

You would be amazed if I told you how many "vital" services are like the blog writers former employers: One button push from destruction.

Oh? Tell me.

Re: How I Fired Myself

#310

Post by geoka9 »

bguthrie wrote: post #205
damm wrote: post #192

Earlier quoted context omitted.

When things like this happen, you have to realize there is more than one 'truth'. There is the truth that here is someone who truncated the users table and because of that it caused the company great harm. Here's another 'truth'. 1. The company lacked backups 2. The junior developer was on a production database. Note: I'm from the oldschool of sysops who feel that you don't give every employee the keys to your kingdo…

The problem isn't who has the keys , it's how they're used . I don't care as much if a junior developer has the prod password; I care more about building an engineering and ops team that understands that dicking around with the prod database isn't okay . Sysops and DBAs are fallible too--I've seen a lot of old school shops that relied heavily on manual migration and configuration. Automate, test, isolate and expect f…

And, most importantly, shake off the "agile" ethos when doing DB migrations. Just forget they exist and triple-check every character you type into the console.
Post reply on HN