Cause of today's Github outage

I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.

Re: Cause of today's Github outage

#7

Post by blutonium » Mon, Nov 15, 2010, 1:59 AM UTC

geophile wrote: post #2

Anyone know what database system they use?

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Re: Cause of today's Github outage

#8

Post by woan » Mon, Nov 15, 2010, 2:04 AM UTC

rookie mistake, better security if they isolate their networks too...

Re: Cause of today's Github outage

#9

Post by jbellis » Mon, Nov 15, 2010, 2:13 AM UTC

blutonium wrote: post #7

geophile wrote: post #2

Anyone know what database system they use?

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

fwiw, Cassandra automatically takes a snapshot (which is basically instantaneous -- see http://www.riptano.com/blog/whats-new-cassandra-066 for a brief explanation why) before truncating or dropping a column family, exactly because it makes recovering from a foot/gun incident like this so much less painful.

Re: Cause of today's Github outage

#10

Post by aaronbrethorst » Mon, Nov 15, 2010, 2:13 AM UTC

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing.

From Chris' Twitter stream (http://twitter.com/#!/defunkt):

Seriously, I blame whoever wrote our crappy continuous integration software.

Oh that's me