Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#111

Post by VeejayRampay »

user5994461 wrote: post #62
notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

it's very bizarre for a python developer like me to read comments like this

uv has been absolutely black magic for everyone in the way it just works and works much faster, but here you are telling us that actually no it's somehow not that good

Re: uv: Deduplicate all files in the wheel cache

#112

Post by Timon3 »

amelius wrote: post #110
Timon3 wrote: post #104

Earlier quoted context omitted.

Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?

You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.

Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox?

Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.

Re: uv: Deduplicate all files in the wheel cache

#113

Post by Xirdus »

Timon3 wrote: post #105
Xirdus wrote: post #76

Earlier quoted context omitted.

True, haven't read the source and only went by what the other commenter said. But one could argue 500MB more free space is similarly imperceptible.

There are still many environments with <=250GB, below that I wouldn't call it negligible. And let's not forget that the price per GB has appreciated a LOT!

They're likely the same environments that can't keep warm cache in memory, so the performance penalties are higher too.

Re: uv: Deduplicate all files in the wheel cache

#114

Post by ketozhang »

insane_dreamer wrote: post #99
zbentley wrote: post #50

Earlier quoted context omitted.

It always could.

not sure it always could. we had to continue using conda for binary packages like cudann, or anything where a wheel wasn't available (nvidia now has Jax-compatible wheels for cuda packages, but it didn't before)

You're asking for something different here. What conda does is package all binary dependencies (e.g., libpq, cuda) that normally should be dynamically linked and provided by your OS.

The pip ecosystem (including uv) doesn't encourage this. Although you could and many have, the build tools don't support it as first class.

For why? Ask what happens if you want an updated version or specific version of CUDA? The library author must add support for runtime linking or users would have rebuild the entire conda package.

Re: uv: Deduplicate all files in the wheel cache

#115

Post by amelius »

Timon3 wrote: post #112
amelius wrote: post #110

Earlier quoted context omitted.

You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.

Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox? Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.

Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?

Re: uv: Deduplicate all files in the wheel cache

#116

Post by globular-toast »

dist-epoch wrote: post #97
globular-toast wrote: post #87

Earlier quoted context omitted.

If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.

> SQLite is a Recommended Storage Format for datasets according to the US Library of Congress. Recommended storage formats are formats which, in the opinion of the preservationists at the Library of Congress, maximizes the chance of survival and continued accessibility of digital content. https://www.sqlite.org/locrsf.html

It also lists XML, JSON and CSV, so they obviously expect you to provide an external checksum. The source even mentions checksums: https://www.loc.gov/preservation/resources/rfs/data.html

Re: uv: Deduplicate all files in the wheel cache

#117

Post by appplication »

marknsikora wrote: post #43
stephenlf wrote: post #27

uv is the backbone of any modern Python library. I’m excited to see improvements. https://stephenlf.dev/blog/python-library-in-2026/

Good article. But the Makefile part should really be replaced by tox as a best practice. Tox even has a uv runner now that will handle setting up all the environments.

Strong disagree on tox. I was all in on it back in 2020-2023 and I found it was a consistent source of headaches and complexity. Whenever we’d run into venv issues it was always tox, often passing or not passing some critical under-the-hood ENV var resulting in reproducibility errors that were difficult to immediately diagnose. It also blurs the line between env vars used to set up tox env and those you want set in your final test env.

We just use containers in CI instead now for matrix testing and it’s so much more clear what is going on with the test env.

Re: uv: Deduplicate all files in the wheel cache

#118

Post by ecnahc515 »

amelius wrote: post #115
Timon3 wrote: post #112

Earlier quoted context omitted.

Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox? Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.

Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?

Well uv runs on many systems and most don't have filesystem level dedupe configured, unfortunately.

Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.

Re: uv: Deduplicate all files in the wheel cache

#119

Post by Timon3 »

amelius wrote: post #115
Timon3 wrote: post #112

Earlier quoted context omitted.

Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox? Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.

Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?

Because (going by scheme271's numbers) I might not want to waste lots of expensive RAM on something or mess around with offline deduplication, when a less resource-intensive alternative is entirely sufficient for whatever my use case is.

Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.

Re: uv: Deduplicate all files in the wheel cache

#120

Post by winstonwinston »

VeejayRampay wrote: post #111
user5994461 wrote: post #62

Earlier quoted context omitted.

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

it's very bizarre for a python developer like me to read comments like this uv has been absolutely black magic for everyone in the way it just works and works much faster, but here you are telling us that actually no it's somehow not that good

I can say that I have used pip only and it does a package manager job well, it just works.

I’m not sure at what uv excel at but reading comments it seems uv is not only package manager but also venv manager.

Post reply on HN