Live data from Hacker News

uv: Deduplicate all files in the wheel cache

github.com

121–126 of 126 posts

Re: uv: Deduplicate all files in the wheel cache

#121

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equ…

Howto use it? sorry for the noob question.

I tried `uv --preview-features content-addressed-cache` and I get an error:

`uv --preview-features content-addressed-cache error: 'uv' requires a subcommand but one was not provided`

Re: uv: Deduplicate all files in the wheel cache

#122
post #80

Earlier quoted context omitted.

Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly. Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have…

> ZFS needs around 1-5GB of ram per TB of storage

...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level).

there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.

Re: uv: Deduplicate all files in the wheel cache

#123

Earlier quoted context omitted.

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly. Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have…

> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level). there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say yo…

Just curious but what would be a good way to design for it ahead of time?

Re: uv: Deduplicate all files in the wheel cache

#124

Earlier quoted context omitted.

> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level). there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say yo…

Just curious but what would be a good way to design for it ahead of time?

mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play.

for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.

Re: uv: Deduplicate all files in the wheel cache

#125
post #13

Earlier quoted context omitted.

> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…

It's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.

“You hate society yet you still participate in it” with smug soyjack face isn’t the slam dunk you think it is.

Re: uv: Deduplicate all files in the wheel cache

#126

Earlier quoted context omitted.

Don't get me wrong, I love `uv` -- I mean as far as being "the backbone of any modern Python library" -- shouldn't any modern python library work with any packing system that follows the PEPs?

It depends on what you mean. The PEPs are essentially a behavioral core, and each tool (build backend, etc.) adds on top of that core. If you’re installing a package, any installer should work; if you’re developing a package, you may be subject to implementation details from your development tool of choice.

"backbone" is what has me hung up, seems more like a scaffold or maybe a bionic exoskeleton.
Post reply on HN