Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How uv got so fast

nesbitt.io

Post reply on HN
Print view View original

411–420 of 468 posts

Re: How uv got so fast

#411

Post by epage »

kibwen wrote: post #203
zahlman wrote: post #66

Earlier quoted context omitted.

> Isn't assigning out what all made things fast presumptive without benchmarks? We also have the benchmark of "pip now vs. pip years ago". That has to be controlled for pip version and Python version, but the former hasn't seen a lot of changes that are relevant for most cases, as far as I can tell. > This also doesn't cover subtle things. Unsure if rkyv is being used to reduce the number of times that TOML is parsed…

> This is interesting in that I wouldn't expect that the typical resolution involves a particularly large quantity of TOML. I don't know the details of Python's resolution algorithm, but for Cargo (which is where epage is coming from) a lockfile (which is encoded in TOML) can be somewhat large-ish, maybe pushing 100 kilobytes (to the point where I'm curious if epage has benchmarked to see if lockfile parsing is notic…

Lockfiles aren't an issue. It is all the dependencies themselves.

Re: How uv got so fast

#412

Post by epage »

TheDong wrote: post #297
pnt12 wrote: post #279

Earlier quoted context omitted.

But once you have a lock file there is no resolution needed, is there? It lists all needed libs and their versions. Given how toml is written, I imagine you can read it incrementally - once a lib section is parsed, you can download it in parallel, even if you didn't parse the whole file yet. (not sure how uv does it, just guessing what can be done)

For whatever it's worth, the toml library uv uses doesn't support streaming parsing: https://github.com/toml-rs/toml/issues/326

TOML as a format doesn't make sense for streaming

- Tables can be in any order, independent of heirarchy

- keys can be dotted, creating subtables in any order

On top of that, most use cases for the format are not benefitted by streaming.

Re: How uv got so fast

#413

Post by VeejayRampay »

andy99 wrote: post #42

I remain baffled about these posts getting excited about uv’s speed. I’d like to see a real poll but I personally can’t imagine people listing speed as one of the their top ten concerns about python package managers. What are the common use cases where the delay due to package installation is at all material? Edit to add: I use python daily

it is absolutely a huge deal, I suspect you work daily in python but with a small number of repositories that dont change too often

Re: How uv got so fast

#414

Post by epage »

pnt12 wrote: post #279
kibwen wrote: post #203

Earlier quoted context omitted.

> This is interesting in that I wouldn't expect that the typical resolution involves a particularly large quantity of TOML. I don't know the details of Python's resolution algorithm, but for Cargo (which is where epage is coming from) a lockfile (which is encoded in TOML) can be somewhat large-ish, maybe pushing 100 kilobytes (to the point where I'm curious if epage has benchmarked to see if lockfile parsing is notic…

But once you have a lock file there is no resolution needed, is there? It lists all needed libs and their versions. Given how toml is written, I imagine you can read it incrementally - once a lib section is parsed, you can download it in parallel, even if you didn't parse the whole file yet. (not sure how uv does it, just guessing what can be done)

For Cargo,

- synchronization operations are implicit so we need to re-resolve to confirm the lockfile is still valid. We could take some short cut but it would require re-implementing some logic

- dependency resolution only uses `Cargo.toml` for local and git dependencies. Registry dependencies have a json summary of what content is relevant for dependency resolution. Cargo parses nearly every locked package's `Cargo.toml` to know how to build it.

Re: How uv got so fast

#415

Post by jacquesm »

zahlman wrote: post #395
jacquesm wrote: post #330

Earlier quoted context omitted.

Scenarios like that occur daily. I do quite a bit of software development and whenever I come across something that really needs python I mentally prepare for a day of battle with the various (all subtly broken) package managers, dependency hell and circular nonsense to the point that I am also ready to give up on it after a day of trying. Just recently: a build of a piece of software that itself wasn't written in py…

> Scenarios like that occur daily. I do quite a bit of software development and whenever I come across something that really needs python I mentally prepare for a day of battle with the various (all subtly broken) package managers, dependency hell and circular nonsense to the point that I am also ready to give up on it after a day of trying. Please name a set of common packages that causes this problem reliably.

You're getting a bit boring, and are not arguing in good faith. "Reliably"... as per your definition I guess. You have now made 60(!!!) comments in this thread questioning everything and everybody without ever once accepting that other people's experiences do not necessarily have to match your own. If you did some reading rather than just writing you'd have seen that I gave a very specific example right in this thread. You are now going on my blocklist because I really don't have time or energy to argue with language zealots.

Re: How uv got so fast

#416

Post by markkitti »

Summary: They fixed Python packaging by not having to run Python to resolve dependencies. Also, they used Rust.

Moral of the story: Use less Python. Use declarative configuration and other langauges instead.

Re: How uv got so fast

#417

Post by oblio »

zahlman wrote: post #76
VerifiedReports wrote: post #36

So... will uv make Python a viable cross-platform utility solution? I was going to learn Python for just that (file-conversion utilities and the like), but everybody was so down on the messy ecosystem that I never bothered.

It has been viable for a long time, and the kinds of projects you describe are likely well served by the standard library.

It hasn't been viable and you'd know if you tried to deploy Python scripts to Windows users and maintain/update them over longer periods of time.

Re: How uv got so fast

#418

Post by Mawr »

zahlman wrote: post #236
crote wrote: post #216

Earlier quoted context omitted.

So basically, it avoids the whole chicken-and-egg problem. With UV you've simply always got "UV -> project Python 1.23 -> project". UV is your dependency manager, and your Python is just another dependency . With other dependency managers you end up with "system Python 3.45 -> dep manager -> project Python 1.23 -> project". Or worse, "system Python 1.23 -> dep manager -> project Python 1.23 -> project". And of course…

Sorry, but that is simply incorrect, on many levels. Virtual environments are the fundamental way of setting up a Python project, whether or not you use uv, which creates and manages them for you . And these virtual environments can freely either use or not use the system environment, whether or not you use uv to create them. It's literally a single-line difference in the `pyvenv.cfg` file, which is a standard requir…

This is more or less the thinking that got us into the mess Python packaging is.

I, as a user, do not care whatsoever about any of this. At all. If you're explaining "virtual environments", you've lost the plot.

Compiled languages got this right. The dev creates a binary and I as a user simply run it. That's it. That's the holy grail.

It's good to see at last someone in the Python space got their ducks in a row and we've finally got a sensible tool.

Re: How uv got so fast

#419

Post by Mawr »

firesteelrain wrote: post #293
zahlman wrote: post #143

Earlier quoted context omitted.

That argument is FUD. The people who created the NPM package manager are not the people who wrote your dependencies. Further, supply chain attacks occur for reasons that are entirely outside NPM's control. Fundamentally they're a matter of trust in the ecosystem — in the very idea of installing the packages in the first place.

Lack of stronger trust controls are part of the larger issue with npm. Pip, Maven and Go are not immune either but they do things structurally better to shift the problem. Go: Enforces global, append-only integrity via a checksum database and version immutability; once a module version exists, its contents cannot be silently altered without detection, shifting attacks away from artifact substitution toward “publish a…

For Go, there are more impactful features: minimal version selection and the culture of fewer, but larger dependencies.

Your average Go project likely has 10x fewer deps than a JS project. Those deps will not get auto-updated to their latest versions either. Much lower attack surface area.

Re: How uv got so fast

#420

Post by rldjbpin »

baby_souffle wrote: post #157
orliesaurus wrote: post #148

The most surprising part of uv's success to me isn't Rust at all, it's how much speed we "unlocked" just by finally treating Python packaging as a well-specified systems problem instead of a pile of historical accidents. If uv had been written in Go or even highly optimized CPython, but with the same design decisions (PEP 517/518/621/658 focus, HTTP range tricks, aggressive wheel-first strategy, ignoring obviously de…

I largely agree but don't want to entirely discount the effect that using a compiled language had. At least in my limited experience, the selling point with the most traction is that you don't already need a working python install to get UV. And once you have UV, you can just go! If I had a dollar for every time I've helped somebody untangle the mess of python environment libraries created by an undocumented mix of p…

> If I had a dollar for every time I've helped somebody untangle the mess of python environment libraries created by an undocumented mix of python delivered through the distributions package management versus native pip versus manually installed...

macos and linux usually come with a python installation out of the box. windows should be following suite but regardless, using uv vs venv is not that different for most users. in fact to use uv in a project, `uv venv` seems like a prerequisite.

Post reply on HN