I think fibers (or, rather, the Async library) in Ruby tends to be fetishized by junior Rails engineers who don't realize higher level thread coordination issues (connection pools, etc) equally apply to fibers. That said this could be a pretty good use case for fibers -- the code base I use every day has ~230 gems and if you can peel off the actual IO bound installation of all those into non-blocking calls, you would see a meaningful performance difference vs spinning up threads and context switching between them.
Can Bundler be as fast as uv?
31–40 of 105 posts
Re: Can Bundler be as fast as uv?
#32I appreciate that Aaron is focusing on the practical algorithm/design improvements that could be made to Bundler, vs. prematurely going all in on "rewrite in Rust".
Speed would be nice, but more than that I want it to also manage Ruby installs. I’m infuriated at the mess of Rubys and version managers.
Mise is the answer to this. I no longer use chruby/rbenv/rvm. And it manages multiple languages, project-local environment, etc.
Re: Can Bundler be as fast as uv?
#33I think fibers (or, rather, the Async library) in Ruby tends to be fetishized by junior Rails engineers who don't realize higher level thread coordination issues (connection pools, etc) equally apply to fibers. That said this could be a pretty good use case for fibers -- the code base I use every day has ~230 gems and if you can peel off the actual IO bound installation of all those into non-blocking calls, you would…
What I would do to really squeeze the rest out in pure ruby (bear in mind I’ve been away about a decade so there _might be_ new bits but nothing meaningful as far as I know):
Use a cheaper to parse index format (the gists I wrote years ago cover this: https://gist.github.com/raggi/4957402)
Use threads for the initial archive downloads (this is just io, and you want to reuse some caches like the index)
Use a few forks for the unpacking and post install steps (because these have unpredictable concurrency behaviors)
Re: Can Bundler be as fast as uv?
#34Earlier quoted context omitted.
> Ye,s but if your CI isn't terrible, you have the dependencies cached, so that subsequent runs are almost instant, and more importantly, you don't have a hard dependency on a third party service. I’d wager the majority of CI usage fits your bill of “terrible”. No provider provides OOTB caching in my experience, and I’ve worked with multiple in house providers, Jenkins, teamcity, GHA, buildkite.
GHA with the `setup-ruby` action will cache gems. Buildkite can be used in tons of different ways, but it's common to use it with docker and build a docker image with a layer dedicated to the gems (e.g. COPY Gemfile Gemfile.lock; RUN bundle install), effectively caching dependencies.
This is what I came to say. We pre cache dependencies into an approved baseline image. And we cache approved and scanned dependencies locally with Nexus and Lifecycle.
Re: Can Bundler be as fast as uv?
#35Well, now my opinion of uv has been damaged. It... > Ignoring requires-python upper bounds. When a package says it requires python Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. > When multiple package indexes are configured, pip checks all of them. uv picks from the first index…
>> Ignoring requires-python upper bounds. When a package says it requires python > Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. Version bound checking is NP complete but becomes tractable by dropping the upper bound constraint. Russ Cox researched version selection in 2016 and…
Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can
1. solve dependency constraints as if upper bounds were absent,
2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then
3. only if the upper bound constraint check fails, fall back to the slower and reliable parser.
This approach would be clever, efficient, and correct. What you don't get to do is just ignore the fucking rules to which another system studiously adheres then claim you're faster than that system.
That's called cheating.
Re: Can Bundler be as fast as uv?
#36I appreciate that Aaron is focusing on the practical algorithm/design improvements that could be made to Bundler, vs. prematurely going all in on "rewrite in Rust".
Speed would be nice, but more than that I want it to also manage Ruby installs. I’m infuriated at the mess of Rubys and version managers.
what exactly is your issue? I've been using rvm for a decade(?) without any major pain. Cross-language tools such as mise or asdf also seem to work ok.
I can relate to the "I wish we didn't need a second tool", but it doesn't seem like much of a mess.
Re: Can Bundler be as fast as uv?
#37Well, now my opinion of uv has been damaged. It... > Ignoring requires-python upper bounds. When a package says it requires python Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. > When multiple package indexes are configured, pip checks all of them. uv picks from the first index…
> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations. Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility. Astral have been very clear that they have no intention of replicating all of pip. uv pip install was a way to smooth the transition from using pip to using uv. The point…
I agree that bytecode compilation (and caching to pyc files) seldom has a meaningful impact, but it's nevertheless unfair to tout it as an advantage of uv over pip, because by doing so you've constructed an apples to oranges comparison.
You could argue that uv has a better default behavior than pip, but that's not an engineering advantage: it's just a different choice of default setting. If you turned off eager bytecode compilation in pip you'd get the same result.
Re: Can Bundler be as fast as uv?
#38Earlier quoted context omitted.
>> Ignoring requires-python upper bounds. When a package says it requires python > Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. Version bound checking is NP complete but becomes tractable by dropping the upper bound constraint. Russ Cox researched version selection in 2016 and…
Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can 1. solve dependency constraints as if upper bounds were absent, 2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then 3. only if the upper bound constraint check fails, fall back to the slower and reliable parser. This approach would b…
It’s not cheating if it works
Re: Can Bundler be as fast as uv?
#39Earlier quoted context omitted.
>> Ignoring requires-python upper bounds. When a package says it requires python > Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. Version bound checking is NP complete but becomes tractable by dropping the upper bound constraint. Russ Cox researched version selection in 2016 and…
Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can 1. solve dependency constraints as if upper bounds were absent, 2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then 3. only if the upper bound constraint check fails, fall back to the slower and reliable parser. This approach would b…
If ignoring the rules makes it faster, then it's still faster. uv has never claimed to be 100% compatible. How often is it actually incorrect?
Re: Can Bundler be as fast as uv?
#40I think fibers (or, rather, the Async library) in Ruby tends to be fetishized by junior Rails engineers who don't realize higher level thread coordination issues (connection pools, etc) equally apply to fibers. That said this could be a pretty good use case for fibers -- the code base I use every day has ~230 gems and if you can peel off the actual IO bound installation of all those into non-blocking calls, you would…
What I would do to really squeeze the rest out in pure ruby (bear in mind I’ve been away about a decade so there _might be_ new bits but nothing meaningful as far as I know): Use a cheaper to parse index format (the gists I wrote years ago cover this: https://gist.github.com/raggi/4957402 ) Use threads for the initial archive downloads (this is just io, and you want to reuse some caches like the index) Use a few fork…
> there _might be_ new bits but nothing meaningful as far as I know
If you didn't need backwards compatibility with older rubies you could use Ractors in lieu of forks and not have to IPC between the two and have cleaner communication channels. I can peg all the cores on my machine with a simple Ractor pool doing simple computation, which feels like a miracle as a Ruby old head. Bundler could get away with creating their own Ractor safe installer pool which would be cool as it'd be the first large scale use of Ractors that I know of.