Earlier quoted context omitted.
> I didn't find it fast at all. Compared to what? Fast is relative. Compared to pip it's miles ahead.
Compared to just running the script with system-wide installs or in a venv, as I mentioned originally
[deleted]
uv: Deduplicate all files in the wheel cache
101–110 of 126 posts
Re: uv: Deduplicate all files in the wheel cache
#102Earlier quoted context omitted.
> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just li…
That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control . The AI cartel is the new Microsoft . It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Intern…
Oh poo, I can download an open weights model now and build glorious things with it. I can train my own, finetune on someone elses, or host a complete package. What is this AI cartel? Are we complaining that they engineer better harnesses? Well, great! What prevents us from doing the same?
Re: uv: Deduplicate all files in the wheel cache
#103Earlier quoted context omitted.
We do a lot more than that, e.g. how to cache distributions is not defined anywhere within PEPs (and should not be, since it’s a purely internal tool decision). But yes, it helps that uv implements the PEP for detached metadata, particularly during resolution.
Don't get me wrong, I love `uv` -- I mean as far as being "the backbone of any modern Python library" -- shouldn't any modern python library work with any packing system that follows the PEPs?
It depends on what you mean. The PEPs are essentially a behavioral core, and each tool (build backend, etc.) adds on top of that core. If you’re installing a package, any installer should work; if you’re developing a package, you may be subject to implementation details from your development tool of choice.
Re: uv: Deduplicate all files in the wheel cache
#104Earlier quoted context omitted.
The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly. Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have…
Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.
Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?
Re: uv: Deduplicate all files in the wheel cache
#105Earlier quoted context omitted.
warm cache case is only imperceptibly slower, no?
True, haven't read the source and only went by what the other commenter said. But one could argue 500MB more free space is similarly imperceptible.
There are still many environments with <=250GB, below that I wouldn't call it negligible. And let's not forget that the price per GB has appreciated a LOT!
Re: uv: Deduplicate all files in the wheel cache
#106> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…
Why BLAKE3? I hadn't thought too deeply about it but at some point I settled on https://github.com/Cyan4973/xxHash for similar things.
Re: uv: Deduplicate all files in the wheel cache
#107> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped…
Why BLAKE3? I hadn't thought too deeply about it but at some point I settled on https://github.com/Cyan4973/xxHash for similar things.
It's probably the fastest strong* cryptographic hash function to date.
Re: uv: Deduplicate all files in the wheel cache
#108Earlier quoted context omitted.
Exactly my experience, even worse because for me it's 2x256GB drives. Eventually I bit the bullet and upgraded one of them to 1TB, but not the one my root partition is mounted on (too lazy for that), so now I find myself moving various large directories across to my "extra" drive and symlinking them. Funnily enough I don't remember this ever being a problem back when 50GB was considered a lot.
Yea, I remember when the family computer had a 40MB hard drive and it was never ever filled to capacity. Now I’m wondering how many new disks to add to my ZFS pool, and when. And how to budget for it.
We upgraded our 40MB harddrive to a 210MB, as it was constantly getting filled up. I was wondering what on earth to do with all this space, it seemed limitless.
A week later I was cleaning up disk space again...
Re: uv: Deduplicate all files in the wheel cache
#109Earlier quoted context omitted.
I think `pdm` is much nicer. I think if pdm had become a think earlier people would have switched from poetry to pdm rather than this uv thing which is a lot of different things at once.
I'm a bit confused. pdm hit 1.0 in 2021, 3 years before uv was released.
It didn't really become that popular. Not as popular as poetry at least. I think the poetry guy famous, but I don't really remember. He was very opinionated, and pdm addressed many of the issues people had with poetry.
Re: uv: Deduplicate all files in the wheel cache
#110Earlier quoted context omitted.
Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.
Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?
You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.