Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PYX: The next step in Python packaging

astral.sh

Post reply on HN
Print view View original

71–80 of 479 posts

Re: PYX: The next step in Python packaging

#71

Post by woodruffw »

zahlman wrote: post #63
woodruffw wrote: post #19

Earlier quoted context omitted.

For what it's worth, I understand this concern. However, I want to emphasize that pyx is intentionally distinct from Astral's tools. From the announcement post: > Beyond the product itself, pyx is also an instantiation of our strategy: our tools remain free, open source, and permissively licensed — forever. Nothing changes there. Instead, we'll offer paid, hosted services that represent the "natural next thing you ne…

Will pyx describe a server protocol that could be implemented by others, or otherwise provide software that others can use to host their own servers? (Or maybe even that PyPI can use to improve its own offering?) That is, when using "paid, hosted services like pyx", is one paying for the ability to use the pyx software in and of itself, or is one simply paying for access to Astral's particular server that runs it?

I might not be following: what would that protocol entail? pyx uses the same PEP 503/691 interfaces as every other Python index, but those interfaces would likely not be immediately useful to PyPI itself (since it already has them).

> or is one simply paying for access to Astral's particular server that runs it?

pyx is currently a service being offered by Astral. So it's not something you can currently self-host, if that's what you mean.

Re: PYX: The next step in Python packaging

#72

Post by poly2it »

giancarlostoro wrote: post #45
m_kos wrote: post #40

> Why is it so hard to install PyTorch, or CUDA, or libraries like FlashAttention or DeepSpeed that build against PyTorch and CUDA? This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience.

Stuff like that led me fully away from Ruby (due to Rails), which is a shame, I see videos of people chugging along with Ruby and loving it, and it looks like a fun language, but when the only way I can get a dev environment setup for Rails is using DigitalOcean droplets, I've lost all interest. It would always fail at compiling something for Rails. I would have loved to partake in the Rails hype back in 2012, but ov…

Have you tried Nix?

https://nixos.org

Re: PYX: The next step in Python packaging

#74

Post by zahlman »

ActorNightly wrote: post #58
monster_truck wrote: post #10

I've been burned too many times by embracing open source products like this. We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place.

I agree. If any of the stuff was worthwhile to pursue, it would be merged into pip.

Pyx represents the server side, not the client side. The analogue in the pre-existing Python world is PyPI.

Many ideas are being added to recent versions of pip that are at least inspired by what uv has done — and many things are possible in uv specifically because of community-wide standards development that also benefits pip. However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). That has a lot to do with why I'm making PAPER.

For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. It remembers entire wheels, but finding them in that cache requires knowing the URL from which they were downloaded. Because PyPI organizes the packages in its own database with its own custom URL scheme, pip would have to reach out to PyPI across the Internet in order to figure out where it put its own downloads!

Re: PYX: The next step in Python packaging

#76

Post by beshrkayali »

dakiol wrote: post #39

I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…

It's not an easy task, and when there's already lots of established practices, habits, and opinions, it becomes even more difficult to get around the various pain points. There's been many attempts: pip (the standard) is slow, lacks dependency resolution, and struggles with reproducible builds. Conda is heavy, slow to solve environments, and mixes Python with non-Python dependencies, which makes understanding some setups very complicated. Poetry improves dependency management but is sluggish and adds unnecessary complexity for simple scripts/projects. Pipenv makes things simpler, but also has the same issue of slow resolution and inconsistent lock files. Those are the ones I've used over the years at least.

uv addressed these flaws with speed, solid dependency resolution, and a simple interface that builds on what people are already used to. It unifies virtual environment and package management, supports reproducible builds, and integrates easily with modern workflows.

Re: PYX: The next step in Python packaging

#78

Post by simonw »

forrestthewoods wrote: post #73

Neat. uv is spectacular. But I don’t get it. How does it work? Why is it able to solve the Python runtime dependency problem? I thought uv had kinda already solved that? Why is a new thingy majig needed?

This is a server hosted SaaS product that acts as a private registry for your company's Python packages.

uv becomes a client that can install those packages from your private registry.

I imagine pip will be able to install them from a PYX registry too.

Re: PYX: The next step in Python packaging

#79

Post by zahlman »

forrestthewoods wrote: post #73

Neat. uv is spectacular. But I don’t get it. How does it work? Why is it able to solve the Python runtime dependency problem? I thought uv had kinda already solved that? Why is a new thingy majig needed?

> Why is it able to solve the Python runtime dependency problem? I thought uv had kinda already solved that?

The dependencies in question are compiled C code that Python interfaces with. Handling dependencies for a graph of packages that are all implemented in pure Python, is trivial.

C never really solved all the ABI issues and especially for GPU stuff you end up having to link against very specific details of the local architecture. Not all of these can be adequately expressed in the current Python package metadata system.

Aside from that, a lot of people would like to have packages that use pre-installed dependencies that came with the system, but the package metadata isn't designed to express those dependencies, and you're also on your own for actually figuring out where they are at runtime, even if you take it on blind faith that the user separately installed them.

This is not really my wheelhouse, though; you'll find a much better write-up at https://pypackaging-native.github.io/ .

Re: PYX: The next step in Python packaging

#80

Post by devjab »

dakiol wrote: post #39

I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…

I work with Python, Node and Go and I don't think any of them have great package systems. Go has an amazing module isolation system and boy do I wish hiding functions within a module/package was as easy in Python as it is in Go. What saves Go is the standard library which makes it possible to write almost everything without needing external dependencies. You've worked with JavaScript and I really don't see how Python is different. I'd argue that Deno and JSR is the only "sane" approach to packages and security, but it's hardly leading and NPM is owned by Microsoft so it's not like you have a great "open source" platform there either. On top of that you have the "fun" parts of ESM vs CommonJS.

Anyway, if you're familiar with Node then I think you can view pip and venv as the npm of Python. Things like Poetry are Yarn, made as replacements because pip sort of sucks. UV on the other hand is a drop-in replacement for pip and venv (and other things) similar to how pnpm is basically npm. I can't answer your question on why there isn't a "good" tool to rule them all, but the single tool has been pip since 2014, and since UV is a drop-in, it's very easy to use UV in development and pip in production.

I think it's reasonable to worry about what happens when Astral needs to make money for their investors, but that's the beauty of UV compared to a lot of other Python tools. It's extremely easy to replace because it's essentially just smarter pip. I do hope Astral succeeds with their pyx, private registries and so on by the way.

Post reply on HN