Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

Post reply on HN
Print view View original

31–40 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#31

Post by embedding-shape »

ashg1-qwzudg wrote: post #29
embedding-shape wrote: post #11

Earlier quoted context omitted.

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control . The AI cartel is the new Microsoft . It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Intern…

> That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.

"Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language.

And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree that big tech sucks, and many places in the world need less surveillance and less corporate control, but I don't think the entire LLM-space (besides very obviously the biggest companies) are responsible for that kind of usage.

Re: uv: Deduplicate all files in the wheel cache

#32

Post by TacticalCoder »

Someone wrote: post #6
TacticalCoder wrote: post #3

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)

Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits).

I wasn't very clear.

Re: uv: Deduplicate all files in the wheel cache

#33

Post by TacticalCoder »

gchamonlive wrote: post #28
TacticalCoder wrote: post #3

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

> I use it for my own "deduplication / integrity / berzerker" utility Do you have it in a public repo you could share?

No sadly I don't have any public repo: it's mostly really a collection of shell scripts and then the database one (for the berzerker/renamer) is in Clojure and I run it from my always-on REPL.

But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.

Re: uv: Deduplicate all files in the wheel cache

#34

Post by blactuary »

stephenlf wrote: post #25
dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

uv is the backbone of any modern Python library. https://stephenlf.dev/blog/python-library-in-2026/

This is exactly the kind of brief summary I was looking for to get my team more organized, thanks for this

Re: uv: Deduplicate all files in the wheel cache

#36

Post by TacticalCoder »

dist-epoch wrote: post #7
TacticalCoder wrote: post #3

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

> It's annoying that most file formats don't checksum their own content.

I agree.

> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.

And backups. So much backups.

Re: uv: Deduplicate all files in the wheel cache

#37

Post by [deleted] »

nchammas wrote: post #23
dboreham wrote: post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

Maybe I've misunderstood you, but that's not a special advantage of uv. pip has been able to do this for many years. pip install git+https://github.com/some-org/repo

[deleted]

Re: uv: Deduplicate all files in the wheel cache

#38

Post by 310298 »

stephenlf wrote: post #27

uv is the backbone of any modern Python library. I’m excited to see improvements. https://stephenlf.dev/blog/python-library-in-2026/

Nonsense. uv is one of many existing build tools. All the items can be done by other programs.

Funny to list OpenAI as an example at the end of the blog post ...

Re: uv: Deduplicate all files in the wheel cache

#39

Post by mminer237 »

CivBase wrote: post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.

Re: uv: Deduplicate all files in the wheel cache

#40

Post by beepbooptheory »

embedding-shape wrote: post #31
ashg1-qwzudg wrote: post #29

Earlier quoted context omitted.

That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control . The AI cartel is the new Microsoft . It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Intern…

> That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control. "Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language. And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree tha…

Is 'tired' here in scare quotes or are you referencing something? And if it wasn't faster from the get-go, how did it address the fatigue?
Post reply on HN