Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

91–100 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#91

Post by MayeulC »

BrokenCogs wrote: post #86
maeln wrote: post #77

Earlier quoted context omitted.

For me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with depen…

For a uv noob like me: what do you mean by bundling it all into a single binary? How does that work?

UV can be built (or downloaded) as a single optionally statically linked with the libc executable file, with no other dependencies. That makes it very easy to distribute on weird systems.

Re: uv: Deduplicate all files in the wheel cache

#92

Post by zanie »

user5994461 wrote: post #69
zanie wrote: post #64

Earlier quoted context omitted.

(I work on uv) > The only advantage of uv is to have support for parallel async extraction. This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. > If pip extracted multiple files/wheels in parall…

> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL. FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code). Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful dura…

> FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).

TIL. I ran some benchmarks and confirmed this is the case for many small files as you'd see in wheels — the GIL is released in a meaningful way for larger files though. Thanks!

> This might not give the same result for uv though.

Yeah, I built a prototype and ran some benchmarks. It makes a big difference if the entire wheel is empty files but for any real world examples it's within noise of the baseline.

Re: uv: Deduplicate all files in the wheel cache

#93

Post by jsmith45 »

charliermarsh wrote: post #54
CivBase wrote: post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Making things faster is much easier than making things smaller. I'm sure we can win back a 4% slowdown elsewhere since we have so many more levers to pull from.

Yeah, my bigger worry here is more about if somebody direct edits a file in their venv and uv was unable to make the venv with reflink/COW semantics and fell back to hardlinks.

This could happen with a user asking their editor to show definition, and then not realizing the result was in some other package, changing it, and saving it, but only if the editor's normal atomic replace mode is disabled. So perhaps that sounds farfetched. But if an AI agent wants to patch some package in a venv, I've seen them bypass the editor and use command line tools to make edits often enough to be concerned. Like corrupting the cache for one package version is already not ideal, and this could potentially affect multiple versions now?

I suppose is this happens the right fix is to nuke the cache completely?

Re: uv: Deduplicate all files in the wheel cache

#94

Post by tingletech »

woodruffw wrote: post #53
tingletech wrote: post #41

Earlier quoted context omitted.

they just implement the modern PEPs

We do a lot more than that, e.g. how to cache distributions is not defined anywhere within PEPs (and should not be, since it’s a purely internal tool decision). But yes, it helps that uv implements the PEP for detached metadata, particularly during resolution.

Don't get me wrong, I love `uv` -- I mean as far as being "the backbone of any modern Python library" -- shouldn't any modern python library work with any packing system that follows the PEPs?

Re: uv: Deduplicate all files in the wheel cache

#95

Post by mpyne »

ashg1-qwzudg wrote: post #29
embedding-shape wrote: post #11

Earlier quoted context omitted.

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control . The AI cartel is the new Microsoft . It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Intern…

> FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.

??? FOSS was around before that, famously so.

The rise of Microsoft made FOSS more important, but FOSS wasn't a reaction to surveillance and control, when it started even multi-user networks were practically unprotected, and it was quite easy indeed to see what other users were up to.

The phone phreaks were the ones who were worried about corporate control of comms, the FOSS people just couldn't figure out why people were paying for software with the hood welded shut when there was better software available to those willing to help contribute to it.

Re: uv: Deduplicate all files in the wheel cache

#96

Post by notatallshaw »

zanie wrote: post #73
optionalsquid wrote: post #71

Earlier quoted context omitted.

> The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?

I haven't done the comparison but it shouldn't be so hard - https://docs.astral.sh/uv/reference/environment/#uv_concurre... - https://docs.astral.sh/uv/reference/environment/#uv_concurre...

Fun fact!

The original request to add a concurrency variable in uv was requested by me because, among other things, I wanted to measure the difference between uv and pip: https://github.com/astral-sh/uv/issues/3311

I'm not really following this performance discussion as I think it's gone off the rails.

Re: uv: Deduplicate all files in the wheel cache

#97

Post by dist-epoch »

globular-toast wrote: post #87
dist-epoch wrote: post #7

Earlier quoted context omitted.

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.

> SQLite is a Recommended Storage Format for datasets according to the US Library of Congress. Recommended storage formats are formats which, in the opinion of the preservationists at the Library of Congress, maximizes the chance of survival and continued accessibility of digital content.

https://www.sqlite.org/locrsf.html

Re: uv: Deduplicate all files in the wheel cache

#98

Post by amelius »

scheme271 wrote: post #88
amelius wrote: post #80

Earlier quoted context omitted.

Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly. Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have…

Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.

Re: uv: Deduplicate all files in the wheel cache

#99

Post by insane_dreamer »

zbentley wrote: post #50
1aj-187 wrote: post #46

Earlier quoted context omitted.

Can uv even be used with C extensions now?

It always could.

not sure it always could. we had to continue using conda for binary packages like cudann, or anything where a wheel wasn't available (nvidia now has Jax-compatible wheels for cuda packages, but it didn't before)

Re: uv: Deduplicate all files in the wheel cache

#100

Post by dimatura »

dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

It's not such a big deal compared to pip for a typical python package, but a couple of use cases where the speed is a nice feature are tool running with uvx and inline dependencies in scripts - you'll still see something is installed the first time you run the tool/script, but usually it's fast enough to not notice.

One could also make a comparison to another popular tool, conda, which is glacially slow, but that would arguably be unfair since conda does things uv doesn't.

More generally, it's still a nice thing, though. It's just a little less friction. Even a few seconds slower can nudge you towards different behavior

Post reply on HN