rookie mistake, better security if they isolate their networks too...
I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.
Cause of today's Github outage
21–30 of 119 posts
Re: Cause of today's Github outage
#22Re: Cause of today's Github outage
#23This will be a good reminder for me to always keep a local copy.
Re: Cause of today's Github outage
#24I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.
I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…
Re: Cause of today's Github outage
#25Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me
Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
Re: Cause of today's Github outage
#26What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.
Re: Cause of today's Github outage
#27Earlier quoted context omitted.
Rails makes this way too easy to do by accident.
If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?
I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests.
Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team.
Many other languages and frameworks don't have it as part of the normal process used by most developers to accidentally do this as much as it has seemed to happen in Rails 2, due to the conventions used.
With Rails, the same project for the most part is in dev, test, and prod. Developers want database.yml revisioned with the rest of the app. Many Rails developers at some point run production against production DB locally, in order to run migrations on production prior to Capistrano deploys, etc., or maybe just to run a production console to run reports or tweak data. So, there is a very real chance for wild things to happen in such a risky environment. I'm not anti-Rails by any means, but I've not heard of this happening as much as it has historically with Rails in other frameworks.
Re: Cause of today's Github outage
#28Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.
Re: Cause of today's Github outage
#29Earlier quoted context omitted.
Rails makes this way too easy to do by accident.
Actually, in Rails 3, there are explicit guards against running the test suite against the production environment. See https://rails.lighthouseapp.com/projects/8994/tickets/5685-t...
So when it's running something like "rake db:reset", which simply recreates all tables & seeds the database, it just looks at your environment and assumes you know what you're doing. Hudson (and other continuous integration systems) are pretty automated so unless it prompted you, it would run the db scripts quietly.