Cause of today's Github outage • Hacker News Board

Re: Cause of today's Github outage

#11

Post by nigelsampson » Mon, Nov 15, 2010, 2:20 AM UTC

aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database.

Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

Re: Cause of today's Github outage

#12

Post by devmonk » Mon, Nov 15, 2010, 2:30 AM UTC

blutonium wrote: post #7

geophile wrote: post #2

Anyone know what database system they use?

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Rails makes this way too easy to do by accident.

Re: Cause of today's Github outage

#13

Post by binaryfinery » Mon, Nov 15, 2010, 2:34 AM UTC

jlangenauer wrote: post #6

I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the tests be run on the production environment. There's bad luck and theres asking for trouble.

Re: Cause of today's Github outage

#14

Post by chunkbot » Mon, Nov 15, 2010, 2:35 AM UTC

I thought systems written in Erlang never go down! ;-)

Re: Cause of today's Github outage

#15

Post by jlangenauer » Mon, Nov 15, 2010, 2:35 AM UTC

devmonk wrote: post #12

blutonium wrote: post #7

Earlier quoted context omitted.

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Rails makes this way too easy to do by accident.

Actually, in Rails 3, there are explicit guards against running the test suite against the production environment.

See https://rails.lighthouseapp.com/projects/8994/tickets/5685-t...

Re: Cause of today's Github outage

#16

Post by gregwebs » Mon, Nov 15, 2010, 2:38 AM UTC

woan wrote: post #8

rookie mistake, better security if they isolate their networks too...

I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.

Re: Cause of today's Github outage

#17

Post by nettdata » Mon, Nov 15, 2010, 2:39 AM UTC

seanmcq wrote: post #4

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments.

We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment.

It's a luxury to have the resources available for this, but it's a life saver, because it really is stupidly easy to make a simple mistake and totally screw things up.

The last time something similar happened to me, it happened to be at the end of a REALLY long day. And what do you know... that day was then made 24 hours longer, interspersed with the occasional cat nap while backups were being restored and verified.

Fun times. Not.

Re: Cause of today's Github outage

#18

Post by blutonium » Mon, Nov 15, 2010, 2:41 AM UTC

aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Context: https://github.com/defunkt/cijoe

Re: Cause of today's Github outage

#19

Post by wccrawford » Mon, Nov 15, 2010, 2:42 AM UTC

nigelsampson wrote: post #11

aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

I did it for the main test database, and STILL felt like crap. SO many headaches.

Re: Cause of today's Github outage

#20

Post by gregwebs » Mon, Nov 15, 2010, 2:43 AM UTC

devmonk wrote: post #12

blutonium wrote: post #7

Earlier quoted context omitted.

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Rails makes this way too easy to do by accident.

If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?