Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ruby deploys temporarily disabled

status.heroku.com

Post reply on HN
Print view View original

61–70 of 98 posts

Re: Ruby deploys temporarily disabled

#61

Post by pilif »

I'm surprised that rubygems.org of all places did not see fit to patch the vulnerability that's now known for multiple weeks and which has been declared to be incredibly dangerous and for which ready-made exploit kits exist.

rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker could deal by uploading backdoored versions of various popular gems.

I know - applying security patches is time-consuming and we are all afraid of breakage. But the moment rubygems.org stepped up to be a semi-official central distribution point for gems, I would have hoped they also took on the responsibility that goes along with that.

If this was some new unknown 0day exploit, I would be much more understanding, but this was known to exist, known to be dangerous, known to be exploited.

Re: Ruby deploys temporarily disabled

#62

Post by saurik »

akavi wrote: post #32

A tangent, but I always thought "YAML" was pronounced /'jæm.ḷ/, however the post's use of "an YAML" suggests it's actually pronounced /waɪ.eɪ.ɛm.ɛl/. Weird.

You still wouldn't say "an why", as "w" isn't a vowel. If Anything, I can understand "an yamel" as more legitimate (as "y" is at least sort of a vowel).

Re: Ruby deploys temporarily disabled

#63

Post by steveklabnik »

pilif wrote: post #61

I'm surprised that rubygems.org of all places did not see fit to patch the vulnerability that's now known for multiple weeks and which has been declared to be incredibly dangerous and for which ready-made exploit kits exist. rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker…

This is not that vulnerability. Their Rails is just fine. This is related to the fact that gems use YAML to store their metadata.

Re: Ruby deploys temporarily disabled

#64

Post by pardner »

pvh wrote: post #56
pardner wrote: post #53

Seems odd, however, that status.heroku.com lists the issue only on the development side, suggesting production apps are not affected? http://screencast.com/t/L36Hpx5dx

Running apps are not affected, but your ability to do development is. The threat vector affects app compilation, not app execution/scaling, which doesn't touch the tainted rubygems.org repositories.

I don't think that is correct. AFAIK the 'development' side of the heroku status panel is not related to developing, per se, it's the status for apps that are not running on production-level resources... for example, single-dyno apps, or apps not running with production flavor of database.

Re: Ruby deploys temporarily disabled

#65

Post by prodigal_erik »

mildavw wrote: post #16

Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline? We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.

We treated .gems like source tarballs (they can't even express non-Ruby dependencies, Gem::Specification#requirements are treated as comments), and only used rubygems.org for newer versions of our dependencies to be packaged as .rpms for test and production deployments. I found that to be much more sane than bypassing the system package manager and smuggling random crap onto dev servers, much less production.

Re: Ruby deploys temporarily disabled

#66

Post by pmahoney »

aneth4 wrote: post #55

This is why you vendor gems, and the current accepted practice of not vendoring gems is dead wrong. I've been chastised before in rails irc for this. I strongly believe the source of all depndencies possible should be in your repository.

> source of all dependencies possible should be in your repository

How far do you go? Do you include libxml for building nokogiri? Heck, do you include libc and gcc for building any gem with a C extension?

Coming from Java, Maven and something like Nexus Sonatype make it easy (for certain values of "easy") to run a proxy repository. The equivalent of all "gem install " goes through the proxy, which continues to serve gems even if the original source goes away.

I don't particularly like the inclusion of dependencies in a repository. Is this a custom version "some guy" long gone from the company created three years ago? Can I safely upgrade it to get security fix ? I suppose similar questions arise no matter the source...

This is reason why you cryptographically sign your gems before publishing them. I (unfortunately) had not known this was supported by RubyGems, but it is: http://docs.rubygems.org/read/chapter/21

But I'll bet very few gems are signed. Rails does not appear to be:

  $ gem install rails -P HighSecurity
  ERROR:  While executing gem ... (Gem::Exception)
    Unsigned gem

Re: Ruby deploys temporarily disabled

#67

Post by bgentry »

pardner wrote: post #64
pvh wrote: post #56

Earlier quoted context omitted.

Running apps are not affected, but your ability to do development is. The threat vector affects app compilation, not app execution/scaling, which doesn't touch the tainted rubygems.org repositories.

I don't think that is correct. AFAIK the 'development' side of the heroku status panel is not related to developing, per se, it's the status for apps that are not running on production-level resources... for example, single-dyno apps, or apps not running with production flavor of database.

You are incorrect: https://devcenter.heroku.com/articles/heroku-status#status-i...

I agree that the production/development split is not entirely clear without additional explanation. We've spent a lot of time thinking about how to communicate these things and have so far not come up with a way that we feel better describes the issues at hand.

Re: Ruby deploys temporarily disabled

#68

Post by timr »

JonnieCache wrote: post #45
Pewpewarrows wrote: post #6

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…

bundle package puts all your app's dependencies in vendor/cache. That can then be put into a git submodule. The problem then becomes the Gemfile and Gemfile.lock, which should really be in that submodule as well. You need to pass flags to bundler commands because it assumes the Gemfile is in the project root.

I don't think Heroku's deploy is smart enough to recognize that you've packaged, right? It'll still try to bundle install, which would break in the current situation.

I think a full solution requires packaging, and using a modified buildpack that skips the bundle step.

Re: Ruby deploys temporarily disabled

#69

Post by 1qaz2wsx3edc »

steveklabnik wrote: post #63
pilif wrote: post #61

I'm surprised that rubygems.org of all places did not see fit to patch the vulnerability that's now known for multiple weeks and which has been declared to be incredibly dangerous and for which ready-made exploit kits exist. rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker…

This is not that vulnerability. Their Rails is just fine. This is related to the fact that gems use YAML to store their metadata.

Semi related CVEs aside (CVE-2013-0333 and CVE-2013-0156): YAML's security problems have been known for years by the community. YAML aside, don't trust user input. This is egg on rubygems's face and the ruby community. I don't think http://rubycentral.org/ has full-time staff for rubygems either.

Gems, specifically should be signed. They are not, this type of exploit will continue to happen, hell, remember when github screwed up ssh keys? Who knows what's in the ecosystem.

TL;DR; Ruby's security ecosystem is butter.

Disclaimer: I love ruby and use it daily. It was two critical problems IMO: unsigned code and GIL. Yes, GIL. I'm looking at you ruby-core.

Re: Ruby deploys temporarily disabled

#70

Post by pardner »

bgentry wrote: post #67
pardner wrote: post #64

Earlier quoted context omitted.

I don't think that is correct. AFAIK the 'development' side of the heroku status panel is not related to developing, per se, it's the status for apps that are not running on production-level resources... for example, single-dyno apps, or apps not running with production flavor of database.

You are incorrect: https://devcenter.heroku.com/articles/heroku-status#status-i... I agree that the production/development split is not entirely clear without additional explanation. We've spent a lot of time thinking about how to communicate these things and have so far not come up with a way that we feel better describes the issues at hand.

Thx for correcting that... have used that screen a million times without realizing they lumped dev apps and prod+dev workflow together. 3 columns would eliminate any confusion (Production Apps, Development Apps, Development Workflow) but might not look at clean.
Post reply on HN