I thought at one point in time rubygems had a system in place to sign the contents of the gem? If not this might be an interesting addition. You could have a digest stored along side every gem file allowing you to validate the authenticity of the gem file... I'm sure others would have something to add to this idea...
Ruby deploys temporarily disabled
11–20 of 98 posts
Re: Ruby deploys temporarily disabled
#12The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.
What is the alternative? You stay on the shared hosting and then what, you get hacked because you didn't verify a gem?
Are the bigwigs going to authorize you time to look through all gems for potential backdoors or are they going to get i for free with their Heroku hosting?
Re: Ruby deploys temporarily disabled
#13I always assumed that Heroku would have an internal proxy for gems. Seems like 80% of users would probably be fetching the same gems that another user might have just fetched. Perhaps something like this could be versioned or snapshotted so that in the event of something like this, you could roll your cache back to that snapshot and let people deploy who had gems in that cache. I'm just thinking out loud.
Which snapshot do you roll back to?
That's a good question. I do know one thing though: I deploy multiple times per day and typically none of my gems have changed.
I guess it would depend on the folks doing the investigation. If an exact timestamp could be determined for when things could have been compromised, you just roll back to a short while before that time.
Re: Ruby deploys temporarily disabled
#14The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.
The exploit still exists whether you are on Heroku, shared or bare metal hosting environment. It's not an issue specific to Heroku, it's an issue that affects ruby gems. Your situation would be worse if you convinced the big wigs to switch to Ruby today.
Re: Ruby deploys temporarily disabled
#15Between this hack and the recent Rails vulnerabilities, it seems like a perfect storm. I wonder if either the hack attempted to tamper with the Rails gems to catch late updaters or to remove the ability to use RubyGems to update to the latest versions and keep vulnerable sites vulnerable.
Re: Ruby deploys temporarily disabled
#16Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline?
We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.
Re: Ruby deploys temporarily disabled
#17Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline? We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.
Gems get yanked. Its a danger of this approach.
Re: Ruby deploys temporarily disabled
#18If you are not updating gems, does it hurt to continue deploying with a custom buildpack? Heroku shouldn't repull gems if the gemspec hasn't been altered. Is that logic correct?
Does anybody have another suggestion for safely working around this issue? I don't have a clear sense for how long this will take to resolve and don't wish to slow down our release pace too much.
Re: Ruby deploys temporarily disabled
#19This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…
They still allow you to deploy you just have to explicitly set a variable in the deploy command so they know you are aware whats going on
Is this safe if you haven't changed any gems since the last deploy? I have a bugfix that I would like to deploy...
Re: Ruby deploys temporarily disabled
#20This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have pre…
Indeed. I presume this is why Perl's package repository CPAN is actually a network of repositories ("Comprehensive Perl Archive Network"); Wikipedia says CPAN "is mirrored worldwide at more than 200 locations."
Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).