How I Fired Myself • Hacker News Board

Re: How I Fired Myself

#321

Post by silverbax88 » Thu, Feb 28, 2013, 4:20 AM UTC

j_baker wrote: post #214

potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

I have to chime in and completely agree. Very lucky. Most people who survive for years at companies have learned to either stay out of sight, or navigate the Treacherous Waters of Blame whenever things go wrong.

This is actually one of the things most employees who have never been managers don't understand.

Re: How I Fired Myself

#322

Post by alxndr » Thu, Feb 28, 2013, 4:30 AM UTC

groby_b wrote: post #233

ajross wrote: post #211

Earlier quoted context omitted.

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Doing that kind of testing is hard . It costs time and effort. If you want to see it done on a truly awe-inspiring scale (whole data centers being taken down by zombies ;) : http://queue.acm.org/detail.cfm?id=2371516

What about Chaos Monkey?

Re: How I Fired Myself

#323

Post by snambi » Thu, Feb 28, 2013, 4:39 AM UTC

This company sucks. You are out of college and doing the first job. Are these stupid enough to give you direct access to production database? if they are making millions in revenue where was there DBAs? Obviously the management got what they deserved. Its unfortunate that it happened through you.

Re: How I Fired Myself

#324

Post by trotsky » Thu, Feb 28, 2013, 4:51 AM UTC

potatolicious wrote: post #124

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

When you think about it it's almost a logical certainty that he'd take the fall. Any company collectively able to understand that the actual failure was inadequate safeguards would have been able to see it coming and presumably would have prevented it from happening. If you're so inexperienced that you expect no one will ever make a mistake you'll obviously assume that the only problem was that someone made one.

It's somewhat fascinating to hear that a company that damaged managed to build a product that actually had users. And I thought my impression of the social gaming segment couldn't go any lower.

Re: How I Fired Myself

#325

Post by derekp7 » Thu, Feb 28, 2013, 4:56 AM UTC

whatshisface wrote: post #204

krallja wrote: post #142

Earlier quoted context omitted.

Neither of those things would have fixed this problem. 1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids" 2. Transaction complete. ... ... "oops!"

Well, to be fair it's harder to type User than it is to miss your rightclick by something that could have only been a few pixels.

Also, I can't tell you how many times I've attempted to click on a button on a web form, and it was still loading and the button moved (along with a different button appearing in its place).

Re: How I Fired Myself

#326

Post by Nikolas0 » Thu, Feb 28, 2013, 4:58 AM UTC

To be honest as a CEO I would fire myself for letting someone in the team work that wrong (I mean in the production server)

Plus there is no excuse for not having backups...

Re: How I Fired Myself

#327

Post by protomyth » Thu, Feb 28, 2013, 5:02 AM UTC

rurounijones wrote: post #317

protomyth wrote: post #102

Earlier quoted context omitted.

At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.

I have done this on a few servers but found that it always screws up formatting of the lines in bash when they are long and you are hitting up and going back through the history. Did you change the $PS1 variable? Can you share your config?

I had it set up on the client side (putty on Windows, terminal on OS X)

Re: How I Fired Myself

#328

Post by monkeyonahill » Thu, Feb 28, 2013, 5:40 AM UTC

That wasn't your failure per se. But the failure of pretty much everyone above you. That they treated you like that after the fact is pretty shitty. In hindsight I'd say that you are much better off by not being there, where you would learn bad practices.

No Stage Environment. Proactively Cancelled Backups on a Business Critical System. Arbitarily implementing features 'because they have it' rather than it having some purpose in the business model. No Test Drills of disaster scenarios. The list goes on. As I say, and you probably realise now, that you are lucky to no longer be there.

Re: How I Fired Myself

#329

Post by deservingend » Thu, Feb 28, 2013, 5:44 AM UTC

hga wrote: post #162

columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…

Seems like sabotage.

Re: How I Fired Myself

#330

Post by SteveGerencser » Thu, Feb 28, 2013, 6:22 AM UTC

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

I worked for a startup a few years back where the CEO deleted the entire 1+TB database/website when the server was hacked and being used as a spam server because he 1. didn't know who to disable the site short of deleting and 2. could reach anyone that did know how.

The next morning he told us to restore the site from backups and fix the security hole. That's when we reminded him, again, that he had refused to pay for backup services for a site of that size.

We all ended up looking fro new jobs withing a couple days.