Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

61–70 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#61

Post by brnt »

embedding-shape wrote: post #11
as1297kj wrote: post #10

[flagged]

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

Maybe you left the L out on purpose, maybe you didn't. In any case: the L is about who is (or is able to) doing the automating. Users, not programmers, should be in charge.

Re: uv: Deduplicate all files in the wheel cache

#62

Post by user5994461 »

notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.

I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.

It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.

It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!

Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks

I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.

Re: uv: Deduplicate all files in the wheel cache

#63

Post by zanie »

Neywiny wrote: post #4
dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but u…

If you want to open an issue with verbose logs (`-vv`) I'd be happy to look into it. You can set `UV_LOG_CONTEXT=1` to emit timing information.

If you're running Python scripts, I'd recommend using `uv lock --script ` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades.

(I work on uv)

Re: uv: Deduplicate all files in the wheel cache

#64

Post by zanie »

user5994461 wrote: post #62
notatallshaw wrote: post #51

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

(I work on uv)

> The only advantage of uv is to have support for parallel async extraction.

This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.

> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.

I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.

> When the user edits one file, all files are modified simultaneously

This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.

> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks

We also explored this (see https://github.com/astral-sh/uv/pull/19694) and the linked pull request has a table comparing to this strategy.

> We have optimization for empty files (0 bytes) because there is nothing to write and checksum.

Interesting, I would be very surprised if this made a significant difference? but I'll take a look.

Re: uv: Deduplicate all files in the wheel cache

#65

Post by Xirdus »

kekebo wrote: post #56
Xirdus wrote: post #52

Earlier quoted context omitted.

When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.

You could reverse the last sentence to "freed disk space reduces the need to clean the hd". And I don't even mean physical limitation as much as the compulsion to manually clean ballooning cache dirs once in a while.

Overprovisioning also reduces the need to clean the disk. I've had a terabyte Macbook for over a year, developing in 7 different tech stacks simultaneously, never cleaning anything up manually, and I still have over 60% space left.

It's a tradeoff between wasting a little time everyday vs. wasting equally little time every year or so but also having the mental load of cleaning up plus maybe existential dread vs. spending more money to never have either problem. I do understand the last one isn't an option for everyone, but if it is, it's absolutely worth it.

Re: uv: Deduplicate all files in the wheel cache

#66

Post by pas »

Xirdus wrote: post #52
mminer237 wrote: post #39

Earlier quoted context omitted.

Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.

When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.

warm cache case is only imperceptibly slower, no?

Re: uv: Deduplicate all files in the wheel cache

#67

Post by zanie »

rjzzleep wrote: post #22
as1297kj wrote: post #10

[flagged]

I think `pdm` is much nicer. I think if pdm had become a think earlier people would have switched from poetry to pdm rather than this uv thing which is a lot of different things at once.

I'm a bit confused. pdm hit 1.0 in 2021, 3 years before uv was released.

Re: uv: Deduplicate all files in the wheel cache

#68

Post by never_inline »

1aj-187 wrote: post #48
never_inline wrote: post #44

Earlier quoted context omitted.

If you're going to be unemployed anyway, and they don't gain a single penny by you using it, then what does this protest gain?

Wrong assumptions: - "unemployed anyway" does not appear in the post you reply to. - They gain market share and goodwill from fools. Not all gains have to be immediate monetary gains. You could as well say that using ChatGPT is beneficial since it causes losses to OpenAI right now.

It's laughable to suggest few people can change OpenAI's destiny by not using uv. Even if Astral goes bottom up, nothing happens OpenAI. So it's not like using ChatGPT.

Re: uv: Deduplicate all files in the wheel cache

#69

Post by user5994461 »

zanie wrote: post #64
user5994461 wrote: post #62

Earlier quoted context omitted.

Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not writ…

(I work on uv) > The only advantage of uv is to have support for parallel async extraction. This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. > If pip extracted multiple files/wheels in parall…

> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.

FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).

Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.

If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.

> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.

Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.

This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D

Re: uv: Deduplicate all files in the wheel cache

#70

Post by Neywiny »

coldtea wrote: post #19
Neywiny wrote: post #4

Earlier quoted context omitted.

I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but u…

> I didn't find it fast at all. Compared to what? Fast is relative. Compared to pip it's miles ahead.

Compared to just running the script with system-wide installs or in a venv, as I mentioned originally
Post reply on HN