Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can Bundler be as fast as uv?

tenderlovemaking.com

Post reply on HN
Print view View original

41–50 of 105 posts

Re: Can Bundler be as fast as uv?

#41

Post by byroot »

maccard wrote: post #30
byroot wrote: post #22

Earlier quoted context omitted.

GHA with the `setup-ruby` action will cache gems. Buildkite can be used in tons of different ways, but it's common to use it with docker and build a docker image with a layer dedicated to the gems (e.g. COPY Gemfile Gemfile.lock; RUN bundle install), effectively caching dependencies.

> GHA with the `setup-ruby` action will cache gems. Caching is a great word - it only means what we want it to mean. My experience with GHA default caches is that it’s absolutely dog slow. > Buildkite can be used in tons of different ways, but it's common to use it with docker and build a docker image with a layer dedicated to the gems (e.g. COPY Gemfile Gemfile.lock; RUN bundle install), effectively caching dependen…

> My experience with GHA default caches is that it’s absolutely dog slow.

GHA is definitely far from the best, but it works:, e.g 1.4 seconds to restore 27 dependencies https://github.com/redis-rb/redis-client/actions/runs/205191...

> The only way docker caching works is if you have a persistent host.

You can pull the cache when the build host spawns, but yes, if you want to build efficiently, you can't use ephemeral builders.

But overall that discussion isn't very interesting because Buildkite is more a kit to build a CI than a CI, so it's on you to figure out caching.

So I'll just reiterate my main point: a CI system must provide a workable caching mechanism if it want to be both snappy and reliable.

I've worked for over a decade on one of the biggest Rails application in existence, and restoring the 800ish gems from cache was a matter of a handful of seconds. And when rubygems.org had to yank a critical gem for copyright reasons [0], we continued building and shipping without disruption while other companies with bad CIs were all sitting ducks for multiple days.

[0] https://github.com/rails/marcel/issues/23

Re: Can Bundler be as fast as uv?

#42

Post by quotemstr »

IshKebab wrote: post #17
quotemstr wrote: post #14

Well, now my opinion of uv has been damaged. It... > Ignoring requires-python upper bounds. When a package says it requires python Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. > When multiple package indexes are configured, pip checks all of them. uv picks from the first index…

> Man, it's easy to be fast when you're wrong. There's never going to be a Python 4 so I don't think they are wrong. Even if lighting strikes thrice there's no way they could migrate people to Python 4 before uv could be updated to "fix" that. > Ambiguity detection is important. I'm not sure what you mean here. Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused…

> Pip doesn't detect any ambiguities. In fact Pip's behaviour is a gaping security hole that they've refused to fix, and as far as I know the only way to avoid it is to use `uv`

Agreed the current behavior is stupid, FWIW. I hope PEPs 708 and 752 get implemented soon. I'm just pointing out that there's an important qualitative difference between

1. we do the same job, but much faster; and

2. we decided your job is stupid and so don't do it, realizing speedups.

uv presents itself as #1 but is actually #2, and that's a shame.

Re: Can Bundler be as fast as uv?

#43

Post by ClikeX »

riffraff wrote: post #36
AlphaSite wrote: post #23

Earlier quoted context omitted.

Speed would be nice, but more than that I want it to also manage Ruby installs. I’m infuriated at the mess of Rubys and version managers.

what exactly is your issue? I've been using rvm for a decade(?) without any major pain. Cross-language tools such as mise or asdf also seem to work ok. I can relate to the "I wish we didn't need a second tool", but it doesn't seem like much of a mess.

Of all the languages I've touched, managing multiple ruby versions has been one of the easiest.

Re: Can Bundler be as fast as uv?

#44

Post by quotemstr »

chuckadams wrote: post #39
quotemstr wrote: post #35

Earlier quoted context omitted.

Perhaps so, although I'm more algorithmically optimistic. If ignoring upper bounds makes the problem more tractable, you can 1. solve dependency constraints as if upper bounds were absent, 2. check that your solution actually satisfies constraints (O(N), quick and passes almost all the time), and then 3. only if the upper bound constraint check fails, fall back to the slower and reliable parser. This approach would b…

If ignoring the rules makes it faster, then it's still faster. uv has never claimed to be 100% compatible. How often is it actually incorrect?

My groceries are cheaper if I walk out of the store without paying for them too. Who's going to stop me?

While I agree that an optimistic optimization for the upper-bound-pass case makes sense, just ignoring the bounds just isn't correct either.

Common pattern in insurgent software is to violate a specification, demonstrate speedups, and then compare yourself favorably to older software that implements the spec (however stupid) faithfully.

Re: Can Bundler be as fast as uv?

#45

Post by [deleted] »

quotemstr wrote: post #44
chuckadams wrote: post #39

Earlier quoted context omitted.

If ignoring the rules makes it faster, then it's still faster. uv has never claimed to be 100% compatible. How often is it actually incorrect?

My groceries are cheaper if I walk out of the store without paying for them too. Who's going to stop me? While I agree that an optimistic optimization for the upper-bound-pass case makes sense, just ignoring the bounds just isn't correct either. Common pattern in insurgent software is to violate a specification, demonstrate speedups, and then compare yourself favorably to older software that implements the spec (howe…

[deleted]

Re: Can Bundler be as fast as uv?

#46

Post by oblio »

quotemstr wrote: post #37
naedish wrote: post #24

Earlier quoted context omitted.

> uv ignores pip’s configuration files entirely. No parsing, no environment variable lookups, no inheritance from system-wide and per-user locations. Stuff like this sense unlikely to contribute to overall runtime, but it does decrease flexibility. Astral have been very clear that they have no intention of replicating all of pip. uv pip install was a way to smooth the transition from using pip to using uv. The point…

I agree that bytecode compilation (and caching to pyc files) seldom has a meaningful impact, but it's nevertheless unfair to tout it as an advantage of uv over pip, because by doing so you've constructed an apples to oranges comparison. You could argue that uv has a better default behavior than pip, but that's not an engineering advantage: it's just a different choice of default setting. If you turned off eager bytec…

> You could argue that uv has a better default behavior than pip, but that's not an engineering advantage: it's just a different choice of default setting. If you turned off eager bytecode compilation in pip you'd get the same result.

Until pip does make the change, this is an engineering advantage for uv. Engineers working on code are part of the product. If I build a car with square wheels and don't change them when I notice the issue, my car still has a bumpy ride, that's a fact.

Re: Can Bundler be as fast as uv?

#47

Post by asa400 »

quotemstr wrote: post #14

Well, now my opinion of uv has been damaged. It... > Ignoring requires-python upper bounds. When a package says it requires python Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice. > When multiple package indexes are configured, pip checks all of them. uv picks from the first index…

> Man, it's easy to be fast when you're wrong. But of course it is fast because Rust not because it just skips the hard parts of dependency constraint solving and hopes people don't notice.

What's underhanded about this? What are the observable effects of this choice that make it wrong? They reformulated the a problem into a different problem that they could solve faster, and then solved that, and got away with it. Sounds like creative problem solving to me.

Re: Can Bundler be as fast as uv?

#48

Post by maccard »

byroot wrote: post #41
maccard wrote: post #30

Earlier quoted context omitted.

> GHA with the `setup-ruby` action will cache gems. Caching is a great word - it only means what we want it to mean. My experience with GHA default caches is that it’s absolutely dog slow. > Buildkite can be used in tons of different ways, but it's common to use it with docker and build a docker image with a layer dedicated to the gems (e.g. COPY Gemfile Gemfile.lock; RUN bundle install), effectively caching dependen…

> My experience with GHA default caches is that it’s absolutely dog slow. GHA is definitely far from the best, but it works:, e.g 1.4 seconds to restore 27 dependencies https://github.com/redis-rb/redis-client/actions/runs/205191... > The only way docker caching works is if you have a persistent host. You can pull the cache when the build host spawns, but yes, if you want to build efficiently, you can't use ephemeral…

> So I'll just reiterate my main point: a CI system must provide a workable caching mechanism if it want to be both snappy and reliable.

The problem is that none of the providers really do this out of the box. GHA kind of does it, but unless you run the runners yourself you’re still pulling it from somewhere remotely.

> I've worked for over a decade on one of the biggest Rails application in existence, and restoring the 800ish gems from cache was a matter of a handful of seconds.

I kind of suspected - the vast majority of orgs don’t have a team of people who can run this kind of a system. Most places with 10-20 devs (which was roughly the size of the team that ran the builds at our last org) have some sort of script, running on cheap as hell runners and they’re not running mirrors and baking base images on dependency changes.

Re: Can Bundler be as fast as uv?

#50

Post by lamontcg »

The biggest thing that gems could do to make rubygems faster is to have a registry/database of files for each gem, so that rubygems didn't have to scan the filesystem on every `require` looking for which gem had which file in it.

That would mean that if you edited your gems directly, things would break. Add a file, and it wouldn't get found until the metadata got rehashed. The gem install, uninstall, etc commands would need to be modified to maintain that metadata. But really, you shouldn't be hacking up your gem library like that ith shellcommands anyway (and if you are doing manual surgery, having to regen the metadata isn't really that burdensome).

Post reply on HN