Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

181–190 of 424 posts

Re: How I Fired Myself

#181

Post by ritchiea »

danso wrote: post #172
paganel wrote: post #140

Earlier quoted context omitted.

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

Heh. Another way I would say that is that "discretion is the better part of valor" And, to rip-off Hitchhiker's Guide to the Galaxy, "cowardice is the better part of discretion"...in that if fear makes you judiciously check your backups and write tests, then that's not a bad thing at all.

I believe developers need the courage to do what we feel is necessary and the paranoia to quadruple check our work and assumptions.

Re: How I Fired Myself

#182

Post by gefh »

hga wrote: post #162
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…

I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...

Re: How I Fired Myself

#183

Post by yaj »

paganel wrote: post #140
ritchiea wrote: post #106

Earlier quoted context omitted.

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

This is the right thing to encourage but I just would like to add always have a backup.

"Don't be afraid to break things as long as you have a backup".

It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.

Re: How I Fired Myself

#184

Post by indrax »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant.

http://www.codinghorror.com/blog/2011/04/working-with-the-ch...

This only happened because nobody even asked "What happens if I press this button?"

Re: How I Fired Myself

#185

Post by narrator »

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

The development environment should not be able to make a direct connection to production. GitHub temporarily deleted their whole prod database because of a config screwup because the dev server could talk to the production db. https://github.com/blog/744-today-s-outage

Re: How I Fired Myself

#186

Post by zulfishah »

Every engineer's worst nightmare. I've worked at a one of the biggest software companies in the world, and I'm working on my own self-funded one-person startup: the panic before doing anything remotely involving production user data is still always nerve-wracking to me. But agree with everyone's assessments here of the failure of the whole company to prevent this. A hardware failure could have just as likely have wiped out all their data. If you're going to cut corners with backing up user data, then you should be prepared to suffer the consequences.

Thanks for sharing this. Took real guts to put it out there.

Re: How I Fired Myself

#188

Post by roryokane »

doktrin wrote: post #174

I found myself doing very much this my very first day on the job working for a software startup. We had a Grails app that acted as a front end for a number of common DB interactions, which were selected via a drop down. One of these (in fact, the default) action was titled "init DB". Of course, this would drop any existing database and initialize a new one. When running through the operational workflow with our COO o…

How did the company deal with the loss of that database? Did they actually have backups, and just restored the data? Did they reconstruct the data from other sources?

Re: How I Fired Myself

#189

Post by munificent »

> The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".

Yes, it is a monumental fuck-up. You put a button in front of a junior developer that can cost the company millions if he accidentally clicks it and doesn't even have undo.

Re: How I Fired Myself

#190

Post by grey-area »

narrator wrote: post #185
grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

The development environment should not be able to make a direct connection to production. GitHub temporarily deleted their whole prod database because of a config screwup because the dev server could talk to the production db. https://github.com/blog/744-today-s-outage

I wasn't suggesting it should, ideally they'd be on completely isolated machines, and there's no reason it has to connect to production. Just because you use production backups to set up your dev environment, doesn't mean the dev environment should be able to talk to production servers, quite the opposite.

What I'd normally do is have a production server, which has daily backups, copies of which are used for dev on local machines, and then pushed to a dev server with separate dev db which is wiped periodically with that production data (a useful test of restoring backups), and has no connection with the production server or db.

Can't work out why they would possibly be doing development on a live db like this, that's insanity.

Post reply on HN