Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ruby deploys temporarily disabled

status.heroku.com

Post reply on HN
Print view View original

21–30 of 98 posts

Re: Ruby deploys temporarily disabled

#21

Post by msmith »

taf2 wrote: post #11

I thought at one point in time rubygems had a system in place to sign the contents of the gem? If not this might be an interesting addition. You could have a digest stored along side every gem file allowing you to validate the authenticity of the gem file... I'm sure others would have something to add to this idea...

It is possible for the publisher to sign the gems [1], but it's not common.

If rubygems.org is keeping fingerprints of each gem, then it still isn't sufficient, since those could have been compromised as well. If there's no other trustworthy source of fingerprints, then maybe we need to crowdsource it. Built a tool that will md5sum all the .gem files in your local cache directory, so that we can look for any files that were changed on rubygems.org

[1] http://docs.rubygems.org/read/chapter/21

Re: Ruby deploys temporarily disabled

#22

Post by stock_toaster »

Pewpewarrows wrote: post #6

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…

  >  It's something that we in the Python community have already learned due to the historical unreliability of our equivalent package repo, PyPI.
learned sounds a touch condescending to me for some reason. The python community has certainly run into it, but (anecdote time) in my experience people still often rely on pypi for their deploys (but use the --mirrors option to pip). Encountered may be more appropriate.

Re: Ruby deploys temporarily disabled

#23

Post by rrikhy »

sergiotapia wrote: post #5

The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.

Feel free to reach out to me if you'd like to have a conversation about how to support your case on Heroku or if you have any questions/concerns; raj@heroku.com

Re: Ruby deploys temporarily disabled

#24

Post by pardner »

sergiotapia wrote: post #5

The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.

From a manager perspective, this sort of alert (as well as recent Heroku emails telling you specifically which app(s) needed patching for the recent Rails CVE issues) is a great example of one of the extra benefits of Heroku. A team of engineers 'watching your back' at no extra charge is a good thing.

Re: Ruby deploys temporarily disabled

#25

Post by benmmurphy »

This is the responsible thing to do. Going through the gems and verifying they aren't compromised is a lot of work. We should be thankful for all the effort the rubygems maintainers and other volunteers are putting into cleaning up this mess.

Re: Ruby deploys temporarily disabled

#26

Post by simonw »

Pewpewarrows wrote: post #6

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…

The way we handle that for our Python deploys is to have a separate "deploy" git repo which includes complete .tar.gz files of all of our dependencies, then have our pip requirements.txt file point to those file paths rather than using external HTTP URLs.

To avoid packages sneakily trying to download their own dependencies from the internet we run pip install with a "--proxy http://localhost:9999 argument (where nothing is actually running on that port) so that we'll see an instant failure if something tries to pull a dependency over the network.

Re: Ruby deploys temporarily disabled

#27

Post by phillmv »

mapgrep wrote: post #20
Pewpewarrows wrote: post #6

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…

Indeed. I presume this is why Perl's package repository CPAN is actually a network of repositories ("Comprehensive Perl Archive Network"); Wikipedia says CPAN "is mirrored worldwide at more than 200 locations." Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).

If I had to guess, I would wager it's because it's expensive and hard. Plus there's the fortunate coincidence that - as far as I recall - rubygems has mostly Just Worked Fine, Thank You Very Much™ .

(I miss the days from when github also hosted a gem repository…)

Solving the authenticity problem alone is probably not fun – tho obviously there is much to be learned from CPAN. Given recent problems there will probably be enough political will to make this happen in the future, though.

Re: Ruby deploys temporarily disabled

#28

Post by rmoriz »

mapgrep wrote: post #20
Pewpewarrows wrote: post #6

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…

Indeed. I presume this is why Perl's package repository CPAN is actually a network of repositories ("Comprehensive Perl Archive Network"); Wikipedia says CPAN "is mirrored worldwide at more than 200 locations." Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).

The central architecture of rubygems allows you to publish and yank gems within minutes. CPAN takes some hours (and deletion may not be controlled).

Personally I'm a big fan of the CPAN approach as it is fairly simply. Just mirror via FTP. It's a nobrainer to setup and run a mirror.

That said, CPAN's master (PAUSE.cpan.org) is a SPOF as well.

What I like is that not a single party is responsible for paying server bills + maintaining the platform. Ruby Central and the team of volunteers do a great job, but in the end, people only care when something breaks.

Instead every big company/university that profits from the Ruby ecosystem should imho run a public rubygems mirror as a contribution to the open source world. That's common practice for other projects, too. Think of all mirrors of the Linux distributions, kernel.org, cpan, python etc.

=> http://slideshare.net/rmoriz/rubygems-behind-the-gems

I also want to mention, that ftp.ruby-lang.org is a single homed box. There is no other official mirror of the MRI/C-Ruby source that can be used as failover or load balancer. This is bad, too.

Re: Ruby deploys temporarily disabled

#29

Post by scotje »

mildavw wrote: post #16

Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline? We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.

If you use bundler, "bundle package" can help reduce or eliminate your dependency on external gem repositories. At least for deployments.

I generally try to follow the "vendor everything" philosophy: http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-a...

Re: Ruby deploys temporarily disabled

#30

Post by rmc »

msmith wrote: post #21
taf2 wrote: post #11

I thought at one point in time rubygems had a system in place to sign the contents of the gem? If not this might be an interesting addition. You could have a digest stored along side every gem file allowing you to validate the authenticity of the gem file... I'm sure others would have something to add to this idea...

It is possible for the publisher to sign the gems [1], but it's not common. If rubygems.org is keeping fingerprints of each gem, then it still isn't sufficient, since those could have been compromised as well. If there's no other trustworthy source of fingerprints, then maybe we need to crowdsource it. Built a tool that will md5sum all the .gem files in your local cache directory, so that we can look for any files th…

Sounds like a good reason to switch to "everything must be signed"
Post reply on HN