Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#81

Post by tedivm »

user5994461 wrote: post #62
notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

There are a ton of advantages to using uv over pip. I don't use UV because it's faster, although I do appreciate that. I use it because it's smart enough to manage virtual environments for each environment and tool, it can isolate to different python versions trivially, and it handles locking in a way that is actual sane.

Re: uv: Deduplicate all files in the wheel cache

#82

Post by notatallshaw »

zanie wrote: post #79
notatallshaw wrote: post #75

Earlier quoted context omitted.

> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios. uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned o…

We didn't compare to pip at the time, but it saved a lot of absolute time for us as reported in the benchmarks from the pull request ( https://github.com/astral-sh/uv/pull/789 ) it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though. But this was just one example optimization, we…

> it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.

This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.

> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.

Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.

Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.

Re: uv: Deduplicate all files in the wheel cache

#83

Post by conradludgate »

coldtea wrote: post #18
OskarS wrote: post #9

Earlier quoted context omitted.

I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not goin…

Because SQLite is a program, that will refuse to start or crash, and which you can trivially replace, if corrupted. Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.

Hopefully it crashes if corrupt... But it could instead jump straight to some code designed to clean up files and delete your database

Re: uv: Deduplicate all files in the wheel cache

#84

Post by zanie »

notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

You might be interested in taking a look at the `uv download` sketch I started on last week https://github.com/astral-sh/uv-dev/pull/875

Re: uv: Deduplicate all files in the wheel cache

#85

Post by notatallshaw »

zanie wrote: post #84
notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

You might be interested in taking a look at the `uv download` sketch I started on last week https://github.com/astral-sh/uv-dev/pull/875

That's excellent news for uv, and I think covers one of two major use cases I often see where uv does not cover standard packaging workflows.

This one being downloading to an offline wheelhouse and installing from that.

The other one being having a shared named global environment ;o).

P.S. I'll have to remove this as an important feature nab has that uv doesn't when I make the announcement nab is no longer experimental, aha.

Re: uv: Deduplicate all files in the wheel cache

#86

Post by BrokenCogs »

maeln wrote: post #77
dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

For me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with depen…

For a uv noob like me: what do you mean by bundling it all into a single binary? How does that work?

Re: uv: Deduplicate all files in the wheel cache

#87

Post by globular-toast »

dist-epoch wrote: post #7
TacticalCoder wrote: post #3

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.

Re: uv: Deduplicate all files in the wheel cache

#88

Post by scheme271 »

amelius wrote: post #80
user5994461 wrote: post #62

Earlier quoted context omitted.

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly.

Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.

Re: uv: Deduplicate all files in the wheel cache

#89

Post by Timon3 »

Zizizizz wrote: post #59
marknsikora wrote: post #43

Earlier quoted context omitted.

Good article. But the Makefile part should really be replaced by tox as a best practice. Tox even has a uv runner now that will handle setting up all the environments.

I like just or mise tasks as often my python projects contain many other language commands (SQL, docker, pnpm, etc ...).

IMO mise tasks are fairly close to the ideal polyglot task runner, because features like argument definitions can be used as progressive enhancements. That means I can write the task logic in shell scripts that can be called without mise, and when I integrate them into mise, I don't have to manually write extra wrappers or duplicate anything. That's my pet peeve with most similar tools, so I can't recommend mise enough!

Re: uv: Deduplicate all files in the wheel cache

#90

Post by [deleted] »

BrokenCogs wrote: post #86
maeln wrote: post #77

Earlier quoted context omitted.

For me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with depen…

For a uv noob like me: what do you mean by bundling it all into a single binary? How does that work?

[deleted]
Post reply on HN