Live data from Hacker News

Cause of today's Github outage

github.com

21–30 of 119 posts

Re: Cause of today's Github outage

#21
post #8

rookie mistake, better security if they isolate their networks too...

I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.

Yeah that is just insane. We typically run prod, stage, test, dev. The test and dev instances are even in another dmz.

Re: Cause of today's Github outage

#24

I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

Re: Cause of today's Github outage

#25

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

I once messed up a table that was responsible for a custom-distributed auto-increment field. If you roll that back and don't reset some servers, you have new data overwriting old data. Very bad stuff.

Re: Cause of today's Github outage

#26
I agree with what's been said so far. 1 - Shit happens 2 - We've all done stupid stuff 3 - Testing environment shouldn't have access to production

What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.

Re: Cause of today's Github outage

#27
post #12

Earlier quoted context omitted.

Rails makes this way too easy to do by accident.

If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?

"how is this Rails specific?"

I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests.

Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team.

Many other languages and frameworks don't have it as part of the normal process used by most developers to accidentally do this as much as it has seemed to happen in Rails 2, due to the conventions used.

With Rails, the same project for the most part is in dev, test, and prod. Developers want database.yml revisioned with the rest of the app. Many Rails developers at some point run production against production DB locally, in order to run migrations on production prior to Capistrano deploys, etc., or maybe just to run a production console to run reports or tweak data. So, there is a very real chance for wild things to happen in such a risky environment. I'm not anti-Rails by any means, but I've not heard of this happening as much as it has historically with Rails in other frameworks.

Re: Cause of today's Github outage

#28

Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.

Since it's git-based, is it even possible to not keep a local copy? I thought everything that goes into github has to be committed to your local git repo first?

Re: Cause of today's Github outage

#29
post #12

Earlier quoted context omitted.

Rails makes this way too easy to do by accident.

Actually, in Rails 3, there are explicit guards against running the test suite against the production environment. See https://rails.lighthouseapp.com/projects/8994/tickets/5685-t...

However in this case, it happened on their continuous integration server -- which probably does more than just "rake test". On our hudson scripts, it does something like "rake db:reset" because some concurrent scripts require data to be persisted in the database and not rolled-back per test.

So when it's running something like "rake db:reset", which simply recreates all tables & seeds the database, it just looks at your environment and assumes you know what you're doing. Hudson (and other continuous integration systems) are pretty automated so unless it prompted you, it would run the db scripts quietly.

Re: Cause of today's Github outage

#30
It kind of makes me wish NILFS2 would become production-ready faster. Give MySQL its own partition, and just roll back to a previous checkpoint if you wipe everything. Not a substitute for backups, but a pretty speedy way to recover for a minor snafu like this.
Post reply on HN