Cause of today's Github outage • Hacker News Board

Re: Cause of today's Github outage

#51

Post by lambda » Mon, Nov 15, 2010, 5:56 AM UTC

SkyMarshal wrote: post #31

nigelsampson wrote: post #11

Earlier quoted context omitted.

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second e…

Heck, you think that's heart attack inducing?

How about the DreamHost case in which they typed the wrong year in their billing code; charged many of their users for an extra year of service, to the tune of $7.5 million:

http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/17/the-final-update/

Re: Cause of today's Github outage

#52

Post by sankara » Mon, Nov 15, 2010, 5:57 AM UTC

May be it's foresight or may be it's just paranoid. We've always used a entirely different username/password in prod and the password for prod never sits in the config files. Kind of saved us a couple of times. Sometimes it doesn't require a highly sophisticated setup to prevent a catastrophe.

Re: Cause of today's Github outage

#53

Post by ammmir » Mon, Nov 15, 2010, 6:04 AM UTC

ammmir wrote: post #39

simply checking if you're talking to a production instance could avert something like this. having some metadata in the db about whether the data stored there is acting as production and at what version and deployment level, so tests can have a sanity check before destructive activities.

why the downvote?

Re: Cause of today's Github outage

#54

Post by seldo » Mon, Nov 15, 2010, 6:06 AM UTC

nigelsampson wrote: post #11

aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who.

It was not a good day.

Re: Cause of today's Github outage

#55

Post by seldo » Mon, Nov 15, 2010, 6:08 AM UTC

ammmir wrote: post #53

ammmir wrote: post #39

simply checking if you're talking to a production instance could avert something like this. having some metadata in the db about whether the data stored there is acting as production and at what version and deployment level, so tests can have a sanity check before destructive activities.

why the downvote?

Presumably because the comment is obvious, unhelpful, and somewhat mean-spirited. I don't think anybody here needs a lesson in how to avoid accidentally dropping your production database.

Re: Cause of today's Github outage

#56

Post by ammmir » Mon, Nov 15, 2010, 6:26 AM UTC

seldo wrote: post #55

ammmir wrote: post #53

Earlier quoted context omitted.

why the downvote?

Presumably because the comment is obvious, unhelpful, and somewhat mean-spirited. I don't think anybody here needs a lesson in how to avoid accidentally dropping your production database.

thanks for the explanation, i didn't mean to be evil. my suggestion might be obvious to some, but definitely not everyone, considering this isn't a singular occurrence.

Re: Cause of today's Github outage

#57

Post by aaronblohowiak » Mon, Nov 15, 2010, 6:31 AM UTC

jrockway wrote: post #47

binaryfinery wrote: post #13

Earlier quoted context omitted.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

So basically, you don't make mistakes, only other people do. Is it cold up there on that high horse?

It is trivial to put up barriers to mistakes.

Re: Cause of today's Github outage

#58

Post by binaryfinery » Mon, Nov 15, 2010, 6:37 AM UTC

jrockway wrote: post #47

binaryfinery wrote: post #13

Earlier quoted context omitted.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

So basically, you don't make mistakes, only other people do. Is it cold up there on that high horse?

No I make mistakes all the time. That's kinda my point.

Re: Cause of today's Github outage

#59

Post by random42 » Mon, Nov 15, 2010, 6:38 AM UTC

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository").

If I were running some critical code, I would have seriously reconsider github, or at-least ask for a detailed explanation on their engineering practices and fail-safe mechanisms.

Re: Cause of today's Github outage

#60

Post by _urga » Mon, Nov 15, 2010, 6:55 AM UTC

I moved my repos off GitHub to my EC2 server a month or two back since they're private and I was only using GitHub for keeping a copy of my code offsite. It's faster for simple push/pull and considering the sunk cost of my EC2 server, also free. I was trying to browse some repos on GitHub yesterday during the downtime and was thankful that my own were still available.