Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

361–370 of 424 posts

Re: How I Fired Myself

#361

Post by pja »

lsc wrote: post #90
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…

I've said it elsewhere, but it bears repeating: RAID is about availability first and foremost. The fact that it happens to preserve your data in the case of one form of hardware failure is a side effect of its primary goal.

Re: How I Fired Myself

#362

Post by ownagefool »

I worked at a small web hosting company that did probably £2m in revenue a year in my first programming job. They had me spending part of my time as support and the other part on projects.

After about 3 or so months they took me took me out of support and literally placed my desk next to the only full time programmer that company had.

They made all changes direct on live servers and I'd already raised this as a concern and now that became my full time job it was agreed that I'd be allowed to create a dev environment.

Long story short, I exported the structure of our MySQL database and imported it into dev. Some variable was wrong so it didn't all import, so I changed the variable, dropped the schema and back to redo.

Yeah that was the live database I just dropped. After a horrible feeling that I can't really explain I fessed up. I dropped it during lunch so it took about two hours to get a restore.

The owner went mad but most other people were sympathetic, telling me their big mistakes and telling me thats what backups were for.

The owner was going crazy about losing money or something and the COO pulled me into a room. I thought I was getting fired but he just asked me what happened and said "yeah we all make mistakes, thats fair enough, just try not to do it again".

I was then told to get on with it and it must have took me a day to finish what would have taken me an hour but I done it and now we had a process and a simple dev environment. I lasted another two years there. I left over money.

Re: How I Fired Myself

#363

Post by sc0rb »

coverband wrote: post #145
potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

Exactly this.

These kind of comments need to be down voted for adding no value.

Re: How I Fired Myself

#364

Post by sdoering »

j_baker wrote: post #214
potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

Oh yes, I second that. Feel really, really lucky...

Re: How I Fired Myself

#365

Post by hef19898 »

protomyth wrote: post #102
hollerith wrote: post #82

Earlier quoted context omitted.

To avoid mistakes like that is why I put the hostname and only the hostname plus one character in my shell prompt. (The other character is a # or $ depending on whether the user is root or not.)

At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.

Interessting, I have been the only guy in my team who had the exact opposite colours. Green was production and red was testing (i didn't do any dev work, so). I guess that came from me being from produciton. But I should have payed more attention when working with other people on my machine, in hindsight... Luckily, nothing bad ever happened!

Re: How I Fired Myself

#366

Post by PeterisP »

ajross wrote: post #244
groby_b wrote: post #233

Earlier quoted context omitted.

Doing that kind of testing is hard . It costs time and effort. If you want to see it done on a truly awe-inspiring scale (whole data centers being taken down by zombies ;) : http://queue.acm.org/detail.cfm?id=2371516

Doing this kind of testing in a gold-plated, heavily-engineered way is hard. But that's not an excuse for not doing it at all. Just walking into your closet and pulling a cable gets you 80-95% of the testing you need, and is free. Setting up a sandbox and "restoring" a backup onto it and then doing some quick queries is likewise easy to do and eliminates huge chunks of the failure space of "bad backups". Really, this…

"walking into your closet and pulling a cable" is not free, if your planned disaster recovery is not a seamless failover, but a process to recover data with some work and limited (nonzero) downtime/cost to business.

For example, our recovery plan for a financial mainframe in case of most major disasters was to restore the daily backup to off-site hardware identical to production hw; however, the (expensive) hardware wasn't "empty" but used as an acceptance test environment.

Doing a full test of the restore would be possible, but it would be a very costly disruption; taking multiple days of work for the actal environment restoration, deployment,testing and then all of this once more to build a proper acceptance-test-environment. Also destroying a few man-months worth of long tests-in-progress and preventing any change deployments while this is happening.

All of this would be reasonable in any real disaster, but such costs and disruptions aren't acceptable for routine testing.

"Chaos Monkey" works only if your infrastucture is built on cheap unstable and massively redundant items. You can also get excellent uptime with expensive, stable, massively controled environment with limited redundancy (100% guaranteed recovery, but not "hot failover") - but you can't afford chaos there.

Re: How I Fired Myself

#367

Post by PeterisP »

paganel wrote: post #140
ritchiea wrote: post #106

Earlier quoted context omitted.

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

Hmm, is it courage like "real men test in production environment" ?

Re: How I Fired Myself

#368

Post by blablabla123 »

silverbax88 wrote: post #321
j_baker wrote: post #214

Earlier quoted context omitted.

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

I have to chime in and completely agree. Very lucky. Most people who survive for years at companies have learned to either stay out of sight, or navigate the Treacherous Waters of Blame whenever things go wrong. This is actually one of the things most employees who have never been managers don't understand.

Your comment makes me think. Are you implying that this is a good practice?

I mean, in fact I do something similar. At our company also a lot of stuff goes wrong. Somehow it surprises me that there was no major fuckup yet. But I do realize that I need to watch out all times that blame never concentrates on me.

It is so easy to blame individuals, it just suffices to have participated somehow in a task that fucked up. Given that all other participants keep a low profile, one needs to learn how to defend/attack in times of blame.

Re: How I Fired Myself

#370

Post by PeterisP »

jiggy2011 wrote: post #199
ismarc wrote: post #104

Earlier quoted context omitted.

Part of our culture/personality/team fit questions is like this. We also have one that's something like "tell me about a time you failed a commitment (deadline, etc) and how you handled it". Numerous people with over a decade claim perfect records, frequently blaming all those around them as having failed. It's been a real easy way to eliminate candidates, particularly because almost every team member we have gave an…

Interesting, though perhaps they misinterpreted the question. Some companies seem to have a culture where you never admit failure and perhaps they assumed that was the answer you were looking for?

Well, that's the whole point - you really don't want to accidentally hire somebody who is "infected" with a culture of never admitting failure.
Post reply on HN