Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

51–60 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#51

Post by notatallshaw »

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.

But it has always had two major issues:

1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")

2. For people with a lot of different environments the cache grows significantly more than pip

I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.

Re: uv: Deduplicate all files in the wheel cache

#52

Post by Xirdus »

mminer237 wrote: post #39
CivBase wrote: post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.

When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.

Re: uv: Deduplicate all files in the wheel cache

#53

Post by woodruffw »

tingletech wrote: post #41
stephenlf wrote: post #27

uv is the backbone of any modern Python library. I’m excited to see improvements. https://stephenlf.dev/blog/python-library-in-2026/

they just implement the modern PEPs

We do a lot more than that, e.g. how to cache distributions is not defined anywhere within PEPs (and should not be, since it’s a purely internal tool decision). But yes, it helps that uv implements the PEP for detached metadata, particularly during resolution.

Re: uv: Deduplicate all files in the wheel cache

#54

Post by charliermarsh »

CivBase wrote: post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Making things faster is much easier than making things smaller. I'm sure we can win back a 4% slowdown elsewhere since we have so many more levers to pull from.

Re: uv: Deduplicate all files in the wheel cache

#55

Post by mark_l_watson »

Nice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use.

I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.

Re: uv: Deduplicate all files in the wheel cache

#56

Post by kekebo »

Xirdus wrote: post #52
mminer237 wrote: post #39

Earlier quoted context omitted.

Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.

When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.

You could reverse the last sentence to "freed disk space reduces the need to clean the hd". And I don't even mean physical limitation as much as the compulsion to manually clean ballooning cache dirs once in a while.

Re: uv: Deduplicate all files in the wheel cache

#57

Post by sharifhsn »

ashg1-qwzudg wrote: post #29
embedding-shape wrote: post #11

Earlier quoted context omitted.

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control . The AI cartel is the new Microsoft . It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Intern…

The Internet relies on physical infrastructure managed by sovereign nations and multinational corporations, it would never be free from those constraints. Until we reach post-scarcity utopia and harmony between all of humanity—ironically, something that could be enabled by AI—the Internet will never be "free".

Re: uv: Deduplicate all files in the wheel cache

#58

Post by drfloyd51 »

NoboruWataya wrote: post #21
colechristensen wrote: post #17

Earlier quoted context omitted.

Whereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage. Disk isn't free, especially now.

Exactly my experience, even worse because for me it's 2x256GB drives. Eventually I bit the bullet and upgraded one of them to 1TB, but not the one my root partition is mounted on (too lazy for that), so now I find myself moving various large directories across to my "extra" drive and symlinking them. Funnily enough I don't remember this ever being a problem back when 50GB was considered a lot.

Because when 50gb was a lot, by the time you filled it, 200 GB was cheap and a non event.

Re: uv: Deduplicate all files in the wheel cache

#59

Post by Zizizizz »

marknsikora wrote: post #43
stephenlf wrote: post #27

uv is the backbone of any modern Python library. I’m excited to see improvements. https://stephenlf.dev/blog/python-library-in-2026/

Good article. But the Makefile part should really be replaced by tox as a best practice. Tox even has a uv runner now that will handle setting up all the environments.

I like just or mise tasks as often my python projects contain many other language commands (SQL, docker, pnpm, etc ...).

Re: uv: Deduplicate all files in the wheel cache

#60

Post by SQLite »

dist-epoch wrote: post #7
TacticalCoder wrote: post #3

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

Checksums use CPU cycles. SQLite will do checksums with an extension (https://sqlite.org/cksumvfs.html) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.
Post reply on HN