Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How uv got so fast

nesbitt.io

Post reply on HN
Print view View original

301–310 of 468 posts

Re: How uv got so fast

#301

Post by Nextgrid »

baby wrote: post #221
Nextgrid wrote: post #158

Earlier quoted context omitted.

It's not just greenfield-ness but the fact it's a commercial endeavor (even if the code is open-source). Building a commercial product means you pay money (or something they equally value) to people to do your bidding. You don't have to worry about politics, licensing, and all the usual FOSS-related drama. You pay them to set their opinions aside and build what you want, not what they want (and if that doesn't work,…

That doesn't make any sense. You can do open source by yourself and not accept any input. How's the company behind uv making money?

> How's the company behind uv making money?

It doesn't have to make money now. But it's clearly pouring commercial-project-level of resources into uv, on the belief they will somehow recoup that investment later on.

Re: How uv got so fast

#303

Post by adrian17 »

firesteelrain wrote: post #287
oefrha wrote: post #242

Earlier quoted context omitted.

Linux systems commonly already provide an outdated system Python you don’t want to use, and it can’t be used to create a venv of a version you want to use. A single Python version for the entire system fundamentally doesn’t work for many people thanks to shitty compat story in the vast ecosystem. Even languages with great compat story are moving to support multi-toolchains natively. For instance, go 1.22 on Ubuntu 24…

Why not just use a Python container rather than rely on having the latest binary installed on the system? Then venv inside the container. That would get you the “venv of a version” that you are referring to

It's more complex and heavier than using uv. I see docker/vm/vagrant/etc as something as something I reach for when the environment I want is too big, too fancy or too nondeterministic to manually set up locally; but the entire point is that "plain Python with some dependencies" really shouldn't qualify as any of these (just like build environment for a random Rust library).

Also, what do you do when you want your to locally test your codebase across many Python versions? Do you keep track of several different containers? If you start writing some tool to wrap that, you're back at square one.

Re: How uv got so fast

#304

Post by oefrha »

zahlman wrote: post #268
oefrha wrote: post #242

Earlier quoted context omitted.

Linux systems commonly already provide an outdated system Python you don’t want to use, and it can’t be used to create a venv of a version you want to use. A single Python version for the entire system fundamentally doesn’t work for many people thanks to shitty compat story in the vast ecosystem. Even languages with great compat story are moving to support multi-toolchains natively. For instance, go 1.22 on Ubuntu 24…

> Linux systems commonly already provide an outdated system Python you don’t want to use Even with LTS Ubuntu updated only at EOL, Python will not be EOL most of the time. > A single Python version for the entire system fundamentally doesn’t work for many people thanks to shitty compat story in the vast ecosystem. My experience has been radically different. Everyone is trying their hardest to provide wheels for a wid…

For much of the ML/scientific ecosystem, you're lucky to get all your deps working with the latest minor version of Python six months to a year after its release. Random ML projects with hundreds to thousands of stars on GitHub may only work with a specific, rather ancient version of Python.

> Because otherwise this problem is trivially solved by anyone competent. In particular, building and installing Python from source is just the standard configure / make / make install dance, and it Just Works. I have done it many times and never needed any help to figure it out even though it was the first thing I tried to build from C source after switching to Linux.

I compiled the latest GCC many times with the standard configure / make / make install dance when I just started learning *nix command line. I even compiled gmp, mpfr, etc. many times. It Just Works. Do you compile your GCC every time before you compile your Python? Why not? It Just Works.

Re: How uv got so fast

#305

Post by aragilar »

orliesaurus wrote: post #148

The most surprising part of uv's success to me isn't Rust at all, it's how much speed we "unlocked" just by finally treating Python packaging as a well-specified systems problem instead of a pile of historical accidents. If uv had been written in Go or even highly optimized CPython, but with the same design decisions (PEP 517/518/621/658 focus, HTTP range tricks, aggressive wheel-first strategy, ignoring obviously de…

Because it broke backwards compatibility? It's worth noting that setuptools is in a similar situation to pip, where any change has a high chance of breaking things (as can be seen by perusing the setuptools and pip bug trackers). PEP 517/518 removed the implementation-defined nature of the ecosystem (which had caused issues for at least a decade, see e.g. the failures of distutils2 and bento), instead replacing it with a system where users complain about which backend to use (which is at least an improvement on the previous situation)...

Re: How uv got so fast

#306

Post by aragilar »

Perseids wrote: post #272
zahlman wrote: post #270

Earlier quoted context omitted.

If they are not developers, it's the developer's responsibility to fix that. The developers have many options available for this.

You misunderstand. The physicists are developing their own software to analyze their experimental data. They typically have little software development experience, but there is seldom someone more knowledgeable available to support them. Making matters worse, they often are not at all interested in software development and thus also don't invest the time to learn more than the absolute minimum necessary to solve thei…

Honestly, they should be using conda (if they're working on their laptops) and the cluster package manager otherwise.

Re: How uv got so fast

#308

Post by aragilar »

Revisional_Sin wrote: post #271

> Ignoring requires-python upper bounds. When a package says it requires python Erm, isn't this a bit bad?

Yes, but it's (probably) the least worse thing they can do given how the "PyPI" ecosystem behaves. As PyPI does not allow replacement of artefacts (sdists, wheels, and older formats), and because there is no way to update/correct metadata for the artefacts, unless the uploader knew at upload time of incompatibilities between their package and and the upper-bounded reference (whether that is the Python interpreter or a Python package), the upper bound does not reflect a known incompatibility. In addition, certain tools (e.g. poetry) added the upper bounds automatically, increasing the amount of spurious bounds. https://iscinumpy.dev/post/bound-version-constraints/ provides more details.

The general lesson from this is when you do not allow changes/replacement of invalid data (which is a legitimate thing to do), then you get stuck with handling the bad data in every system which uses it (and then you need to worry about different components handling the badness in different ways, see e.g. browsers).

Re: How uv got so fast

#309

Post by raverbashing »

Nextgrid wrote: post #158
orliesaurus wrote: post #148

The most surprising part of uv's success to me isn't Rust at all, it's how much speed we "unlocked" just by finally treating Python packaging as a well-specified systems problem instead of a pile of historical accidents. If uv had been written in Go or even highly optimized CPython, but with the same design decisions (PEP 517/518/621/658 focus, HTTP range tricks, aggressive wheel-first strategy, ignoring obviously de…

It's not just greenfield-ness but the fact it's a commercial endeavor (even if the code is open-source). Building a commercial product means you pay money (or something they equally value) to people to do your bidding. You don't have to worry about politics, licensing, and all the usual FOSS-related drama. You pay them to set their opinions aside and build what you want, not what they want (and if that doesn't work,…

I 100% agree with this

And it's true, while I disagree with a lot of systemd decisions focus has a leveraging effect that's disproportional

Re: How uv got so fast

#310

Post by carderne »

zahlman wrote: post #150
woodruffw wrote: post #144

Earlier quoted context omitted.

> Yeah, so you'd have to pass around the `BytesIO` instead. That wouldn’t be zero-copy either: BytesIO is an I/O abstraction over a buffer, so it intentionally masks the “lifetime” of the original buffer. In effect, reading from the BytesIO creates new copies of the underlying data by design, in new `bytes` objects. (This is actually a great capsule example of why zero-copy design is difficult in Python: the Pythonic…

Fair. (You can `.getbuffer` but you still have to keep the underlying BytesIO object "open" somehow.) I'm not convinced this is going to bottleneck things, though. (On the flip side, I guess the OS is likely to cache any disk write in memory anyway.)

I’m just a casual observer of this thread, but I think you’d find it worthwhile to read up a bit on zero-copy stuff.

It’s ~impossible in Python (because you don’t control memory) and hard in C/similar (because of use-after-free).

Rust’s borrow checker makes it easier, but it’s still tricky (for non-trivial applications). You have to do all your transformations and data movements while only referencing the original data.

Post reply on HN