Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

11–20 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#11

[flagged]

> By using uv, you are literally supporting the people who want to make you unemployed for stock options.

By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.

Re: uv: Deduplicate all files in the wheel cache

#12

[flagged]

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

What is the word? "Facetious"?

Re: uv: Deduplicate all files in the wheel cache

#13

[flagged]

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

It's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.

Re: uv: Deduplicate all files in the wheel cache

#15

> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…

Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.

Re: uv: Deduplicate all files in the wheel cache

#17
post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Whereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage.

Disk isn't free, especially now.

Re: uv: Deduplicate all files in the wheel cache

#18
post #9

Earlier quoted context omitted.

It's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as fil…

I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not goin…

Because SQLite is a program, that will refuse to start or crash, and which you can trivially replace, if corrupted.

Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.

Re: uv: Deduplicate all files in the wheel cache

#19
post #4
post #2

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but u…

>I didn't find it fast at all.

Compared to what? Fast is relative. Compared to pip it's miles ahead.

Re: uv: Deduplicate all files in the wheel cache

#20
post #16

A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.

Whereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage. Disk isn't free, especially now.

I don't disagree. But if you're in that position, surely you'd be better off emptying the cache and disabling it?
Post reply on HN