uv: Deduplicate all files in the wheel cache • Hacker News Board

Re: uv: Deduplicate all files in the wheel cache

#71

Post by optionalsquid » Mon, Aug 31, 2026, 3:51 PM UTC

user5994461 wrote: post #62

notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

> The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.

That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?

Re: uv: Deduplicate all files in the wheel cache

#72

Post by pama » Mon, Aug 31, 2026, 3:54 PM UTC

TacticalCoder wrote: post #32

Someone wrote: post #6

Earlier quoted context omitted.

For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)

Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits). I wasn't very clear.

So at 3 million different files you have a 98.3% chance of a hash collision. Wouldnt that cause problems in real datasets?

Re: uv: Deduplicate all files in the wheel cache

#73

Post by zanie » Mon, Aug 31, 2026, 3:56 PM UTC

optionalsquid wrote: post #71

user5994461 wrote: post #62

Earlier quoted context omitted.

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

> The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?

I haven't done the comparison but it shouldn't be so hard

- https://docs.astral.sh/uv/reference/environment/#uv_concurre...

Re: uv: Deduplicate all files in the wheel cache

#74

Post by dist-epoch » Mon, Aug 31, 2026, 3:58 PM UTC

SQLite wrote: post #60

dist-epoch wrote: post #7

Earlier quoted context omitted.

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

Checksums use CPU cycles. SQLite will do checksums with an extension ( https://sqlite.org/cksumvfs.html ) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.

Thank you for creating it!

I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.

FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).

Re: uv: Deduplicate all files in the wheel cache

#75

Post by notatallshaw » Mon, Aug 31, 2026, 4:00 PM UTC

zanie wrote: post #64

user5994461 wrote: post #62

Earlier quoted context omitted.

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

(I work on uv) > The only advantage of uv is to have support for parallel async extraction. This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. > If pip extracted multiple files/wheels in parall…

> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.

As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.

uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:

* We did a lot of work to make version parsing twice as fast: https://iscinumpy.dev/post/packaging-faster/

* Since that blog post I made typical version parse three times faster on top of that: https://github.com/pypa/packaging/pull/1082

* Also since that blog post version filtering has gone through multiple optimizations and in some cases will be more than 30x faster e.g. https://github.com/pypa/packaging/pull/1105, https://github.com/pypa/packaging/pull/1111, https://github.com/pypa/packaging/pull/1120

At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).

Re: uv: Deduplicate all files in the wheel cache

#76

Post by Xirdus » Mon, Aug 31, 2026, 4:06 PM UTC

pas wrote: post #66

Xirdus wrote: post #52

Earlier quoted context omitted.

When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.

warm cache case is only imperceptibly slower, no?

True, haven't read the source and only went by what the other commenter said. But one could argue 500MB more free space is similarly imperceptible.

Re: uv: Deduplicate all files in the wheel cache

#77

Post by maeln » Mon, Aug 31, 2026, 4:08 PM UTC

dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

For me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with dependencies.

None of it is unique to `uv` I believe, but it does it all, reliably, is easy to install, and easy to use. In terms of DX, for my use cases, it beat poetry, pipenv, pyenv (for python version management) and just using pip

Re: uv: Deduplicate all files in the wheel cache

#78

Post by jvolkman » Mon, Aug 31, 2026, 4:09 PM UTC

user5994461 wrote: post #62

notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

> The only advantage of uv is to have support for parallel async extraction

Maybe the only advantage in a particular area (installing)? Because there are many other advantages.

The rich lock file for instance allows for much better cross-platform tooling. I can build a linux Docker container from a macos build host, for example, without VMs or any other emulation - simply using the cross-platform details in uv.lock and the correct tooling. I can even cross-compile numpy and other native wheels (linux -> macos, macos -> linux).

Other locker tools provide similar cross-platform information (Poetry, PDM), but pip is still lacking.

Re: uv: Deduplicate all files in the wheel cache

#79

Post by zanie » Mon, Aug 31, 2026, 4:09 PM UTC

notatallshaw wrote: post #75

zanie wrote: post #64

Earlier quoted context omitted.

(I work on uv) > The only advantage of uv is to have support for parallel async extraction. This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. > If pip extracted multiple files/wheels in parall…

> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios. uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned o…

We didn't compare to pip at the time, but it saved a lot of absolute time for us as reported in the benchmarks from the pull request (https://github.com/astral-sh/uv/pull/789) it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.

But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)

Re: uv: Deduplicate all files in the wheel cache

#80

Post by amelius » Mon, Aug 31, 2026, 4:21 PM UTC

user5994461 wrote: post #62

notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.