Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

241–250 of 424 posts

Re: How I Fired Myself

#241

Post by reinhardt »

happycube wrote: post #225
downandout wrote: post #223

Earlier quoted context omitted.

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have…

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

Re: How I Fired Myself

#242

Post by fitandfunction »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Exactly. Problems with production databases are inevitable. It's just a matter of time.

The guy who should be falling on the sword, if anyone, is the person in charge of backups.

Better yet, the CEO or CTO should have made this a learning opportunity and taken the blame for the oversight + praised the team for banding together and coming up with a solution + a private chat with the OP.

Re: How I Fired Myself

#243

Post by sixtypoundhound »

yaj wrote: post #183
paganel wrote: post #140

Earlier quoted context omitted.

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

This is the right thing to encourage but I just would like to add always have a backup. "Don't be afraid to break things as long as you have a backup". It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.

I would go one step farther....

Every production deployment should involve blowing away the prior instance, rebuilding from scratch, and restarting the service; you are effectively doing a near-full "restore" for every deployment, which forces you to have everything fully backed up and accessible...

Any failure to maintain good business continuity practices will manifest early for a product / employee / team, which allows you to prevent larger failures...

Re: How I Fired Myself

#244

Post by ajross »

groby_b wrote: post #233
ajross wrote: post #211

Earlier quoted context omitted.

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Doing that kind of testing is hard . It costs time and effort. If you want to see it done on a truly awe-inspiring scale (whole data centers being taken down by zombies ;) : http://queue.acm.org/detail.cfm?id=2371516

Doing this kind of testing in a gold-plated, heavily-engineered way is hard. But that's not an excuse for not doing it at all. Just walking into your closet and pulling a cable gets you 80-95% of the testing you need, and is free. Setting up a sandbox and "restoring" a backup onto it and then doing some quick queries is likewise easy to do and eliminates huge chunks of the failure space of "bad backups".

Really, this attitude (that things have to be done right) is part of the problem here. To a seasoned IT wonk, the only alternative to doing something "The Right Way" is not doing it at all. And that's a killer in situations like these.

Don't hack your systems to make them work. Absolutely do hack at them to test.

Re: How I Fired Myself

#245

Post by georgemcbay »

reinhardt wrote: post #241
happycube wrote: post #225

Earlier quoted context omitted.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

Unfortunately it sounds perfectly plausible to me.

"Why are we paying for backups? The database has never failed (yet)"!

Re: How I Fired Myself

#246

Post by joecurry »

lsc wrote: post #90
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…

>> I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it.

I test backups for F500 companies on a daily basis (IT Risk Consulting) - this would be missing the point really, the business process around this problem is really moving towards live mirrored replication. This allows much faster recall time, and also mitigates many risks with the conventional 'snapshot' method through either tapes, cloud, etc.

Re: How I Fired Myself

#248

Post by fusiongyro »

Qantourisc wrote: post #237
fusiongyro wrote: post #37

Earlier quoted context omitted.

It's a nice theory, but not everybody has that much staff. You can get by with proper backups and running against local copies of production.

Constrains also act like bug protections. If you screw up your database tells you right away, and not a beta tester much later.

You're preaching to the choir. :)

Re: How I Fired Myself

#249

Post by KenL »

I agree with the comments here that spread the blame past this author.

I manage a large number of people at a news .com site and know that screw-ups are always a combination of two factors: people & systems.

People are human and will make mistakes. We as upper management have to understand that and create systems, of various tolerance, that deal with those mistakes.

If you're running a system allowing a low-level kid to erase your data, that was your fault.

I'd never fire someone for making a stupid mistake unless it was a pattern.

Re: How I Fired Myself

#250

Post by jdmaresco »

I had a similar situation when collaborating with a team on a video project during a high school internship. Somehow I managed to delete the entire timeline accounting for hours of editing work that my boss had put in. To this day I don't know how it happened, I just looked down and all the clips were gone from the timeline. In the end, I think we found some semblance of a backup, and at least we didn't lose the raw data/video content, but I can relate to the out-of-body experience that hits you when you realize you just royally screwed up your team's progress and there's nothing you can do about it.
Post reply on HN