Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How uv got so fast

nesbitt.io

Post reply on HN
Print view View original

131–140 of 468 posts

Re: How uv got so fast

#131

Post by zahlman »

skywhopper wrote: post #32

This is great to read because it validates my impression that Python packaging has always been a tremendous overengineered mess. Glad to see someone finally realized you just need a simple standard metadata file per package.

It has been realized in the Python community for a very long time. But there have been years of debate over the contents and formatting, and years of trying to figure out how to convince authors and maintainers to do the necessary work on their end, and years of trying to make sure the ecosystem doesn't explode from trying to remove legacy support.

There are still separate forms of metadata for source packages and pre-compiled distributions. This is necessary because of all the weird idiosyncratic conditional logic that might be necessary in the metadata for platform-specific dependencies. Some projects are reduced to figuring out the final metadata at build time, while building on the user's machine, because that's the only way to find out enough about the user's machine to make everything work.

It really isn't as straightforward as you'd expect, largely because Python code commonly interfaces to compiled code in several different languages, and end users expect this to "just work", including on Windows where they don't have a compiler and might not know what that is.

See https://pypackaging-native.github.io/ for the general flavour of it.

Re: How uv got so fast

#133

Post by zahlman »

woodruffw wrote: post #116
zahlman wrote: post #106

Earlier quoted context omitted.

I can't even imagine what "safety" issue you have in mind. Given that "zero-copy" apparently means "in-memory" (a deserialized version of the data necessarily cannot be the same object as the original data), that's not even difficult to do with the Python standard library. For example, `zipfile.ZipFile` has a convenience method to write to file, but writing to in-memory data is as easy as with zipfile.ZipFile(archive…

> Given that "zero-copy" apparently means "in-memory" (a deserialized version of the data necessarily cannot be the same object as the original data), that's not even difficult to do with the Python standard library This is not what zero-copy means. Here's a working definition[1]. Specifically, it's not just about keeping things in memory; copying in memory is normal. The goal is to not make copies (or more precisely…

> Yes, that's what makes it not zero-copy.

Yeah, so you'd have to pass around the `BytesIO` instead.

I know that zero-copy doesn't ordinarily mean what I described, but that seemed to be how TFA was using it, based on the logic in the rest of the sentence.

Re: How uv got so fast

#134

Post by BiteCode_dev »

Other design decisions that made uv fast:

- uncompressing packages while they are still being downloaded, in memory, so that you only have to write to disk once

- design of its own locking format for speed

But yes, rust is actually making it faster because:

- real threads, no need for multi-processing

- no python VM startup overhead

- the dep resolution algo is exactly the type of workload that is faster in a compiled language

Source, this interview with Charlie Marsh: https://www.bitecode.dev/p/charlie-marsh-on-astral-uv-and-th...

The guy has a lot of interesting things to say.

Re: How uv got so fast

#135

Post by zahlman »

KPGv2 wrote: post #115
zahlman wrote: post #97

Earlier quoted context omitted.

> having relatively curious, meticulous, detail-oriented people on hand who were interested in solving hard problems.... Had we built those projects with JavaScript or Python I suspect the outcomes would have been worse for reasons apart from the language choice. I genuinely can't understand why you suppose that has to do with the implementation language at all.

> I genuinely can't understand why you suppose that has to do with the implementation language at all. Languages that attract novice programmers (JS is an obvious one; PHP was one 20 years ago) have a higher noise to signal ratio than one that attracts intermediate and above programmers. If you grabbed an average Assembly programmer today, and an average JavaScript programmer today, who do you think is more careful a…

Sure, but that kind of incompetence is already filtered out (in the https://www.lesswrong.com/w/screening-off-evidence sense) by the task of creating a package installer.

Re: How uv got so fast

#137

Post by hk1337 »

It’s fast because it sucks the life force from bad developers to make them into something good.

Jokes aside…

I really like uv but also really like mise and I cannot seem to get them to work well together.

Re: How uv got so fast

#138

Post by IgorPartola »

steve_adams_86 wrote: post #68
glaslong wrote: post #60

Earlier quoted context omitted.

Someone once told me a benefit of staffing a project for Haskell was it made it easy to select for the types of programmers that went out of their way to become experts in Haskell. Tapping the Rust community is a decent reason to do a project in Rust.

I'm my experience this is definitely where rust shined. The language wasn't really what made the project succeed so much as having relatively curious, meticulous, detail-oriented people on hand who were interested in solving hard problems. Sometimes I thought our teams would be a terrible fit for more cookie-cutter applications where rapid development and deployment was the primary objective. We got into the weeds al…

Rust is also a systems language. I am still wrapping my mind around why it is so popular for so many end projects when its main use case and goals were basically writing a browser a maybe OS drivers.

But that’s precisely why it is good for developer tools. And it turns out people who write systems code are really damn good at writing tools code.

As someone who cut my teeth on C and low level systems stuff I really ought to learn Rust one of these days but Python is just so damn nice for high level stuff and all my embedded projects still seem to require C so here I am, rustless.

Re: How uv got so fast

#139

Post by zzzeek »

> pip could implement parallel downloads, global caching, and metadata-only resolution tomorrow. It doesn’t, largely because backwards compatibility with fifteen years of edge cases takes precedence. But it means pip will always be slower than a tool that starts fresh with modern assumptions.

what does backwards compatibility have to do with parallel downloads? or global caching? The metadata-only resolution is the only backwards compatible issue in there and pip can run without a setup.py file being present if pyproject.toml is there.

Short answer is most, or at least a whole lot, of the improvements in uv could be integrated into pip as well (especially parallelizing downloads). But they're not, because there is uv instead, which is also maintained by a for-profit startup. so pip is the loser

Re: How uv got so fast

#140

Post by IgorPartola »

zahlman wrote: post #135
KPGv2 wrote: post #115

Earlier quoted context omitted.

> I genuinely can't understand why you suppose that has to do with the implementation language at all. Languages that attract novice programmers (JS is an obvious one; PHP was one 20 years ago) have a higher noise to signal ratio than one that attracts intermediate and above programmers. If you grabbed an average Assembly programmer today, and an average JavaScript programmer today, who do you think is more careful a…

Sure, but that kind of incompetence is already filtered out (in the https://www.lesswrong.com/w/screening-off-evidence sense) by the task of creating a package installer.

You would think so, yet here I am sitting with a node_modules full of crud placed there by npm, waiting for the next supply chain attack.
Post reply on HN