i wonder if nix has been considered
PYX: The next step in Python packaging
81–90 of 479 posts
Re: PYX: The next step in Python packaging
#82Earlier quoted context omitted.
Will pyx describe a server protocol that could be implemented by others, or otherwise provide software that others can use to host their own servers? (Or maybe even that PyPI can use to improve its own offering?) That is, when using "paid, hosted services like pyx", is one paying for the ability to use the pyx software in and of itself, or is one simply paying for access to Astral's particular server that runs it?
I might not be following: what would that protocol entail? pyx uses the same PEP 503/691 interfaces as every other Python index, but those interfaces would likely not be immediately useful to PyPI itself (since it already has them). > or is one simply paying for access to Astral's particular server that runs it? pyx is currently a service being offered by Astral. So it's not something you can currently self-host, if…
> pyx uses the same PEP 503/691 interfaces as every other Python index
... Then how can it make decisions about how to serve the package request that PyPI can't? Is there not some extension to the protocol so that uv can tell it more about the client system?
Re: PYX: The next step in Python packaging
#83The real thing that I hope someone is able to solve is downloading such huge amounts of unnecessary code. As I understand, the bulk of the torch binary is just a huge nvfatbin compiled for every SM under the sun when you usually just want it to run on whatever accelerators you have on hand. Even just making narrow builds of like `pytorch-sm120a` (with stuff like cuBLAS thin binaries paired with it too) as part of a h…
Another piece is that PyPI has no index— it's just a giant list of URLs [1] where any required metadata (eg, the OS, python version, etc) is encoded in the filename. That makes it trivial to throw behind a CDN since it's all super static, but it has some important limitations:
- there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info)
- there's no way to get hashes of the archives
- there's no way to do things like reverse-search (show me everything that depends on x)
I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc.
Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it.
[1]: Top-level: https://pypi.org/simple/ Individual package: https://pypi.org/simple/pyyaml/
Re: PYX: The next step in Python packaging
#84I've been burned too many times by embracing open source products like this. We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place.
I agree. If any of the stuff was worthwhile to pursue, it would be merged into pip.
It’s not that complex - just try it
Re: PYX: The next step in Python packaging
#85> Why is it so hard to install PyTorch, or CUDA, or libraries like FlashAttention or DeepSpeed that build against PyTorch and CUDA? This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience.
This was basically the reason to use anaconda back in the day.
In my experience, Anaconda (including Miniconda, Micromamba, IntelPython, et al.) is still the default choice in scientific computing and machine learning.
Re: PYX: The next step in Python packaging
#86Earlier quoted context omitted.
> Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/ ). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then. Python packaging is (largely) solving problems t…
As someone outside those communities, could you elaborate?
Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).
I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS? So that would be one answer to what Python packaging is trying to solve that node isn't trying to address.
Re: PYX: The next step in Python packaging
#87I wonder whether it will have a flat namespace that everyone competes over or whether the top-level keys will be user/project identifiers of some sort. I hope the latter.
I feel like github solved this problem pretty well.
Re: PYX: The next step in Python packaging
#88Earlier quoted context omitted.
I think you are making a good point, but please don't use the old Steve Baller FUD term, "viral." Copyleft is a better term
The word "left" is now very charged too, maybe even more than "viral".
Every word is charged now, so you might as well use it. "Copyleft" is a fine pun on "copyright".
Re: PYX: The next step in Python packaging
#89Earlier quoted context omitted.
For what it's worth, I understand this concern. However, I want to emphasize that pyx is intentionally distinct from Astral's tools. From the announcement post: > Beyond the product itself, pyx is also an instantiation of our strategy: our tools remain free, open source, and permissively licensed — forever. Nothing changes there. Instead, we'll offer paid, hosted services that represent the "natural next thing you ne…
> Basically, we're hoping to address this concern by building a separate sustainable commercial product rather than monetizing our open source tools. jfrog artifactory suddenly very scared for its safety
Only a matter of time before someone makes something better than Artifactory. It’s a low bar to hit imho.
Re: PYX: The next step in Python packaging
#90I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…
I appreciate everything they’ve done but the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity.
It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem.
I suspect it’s also a misalignment of interests. No one there really invests in improving UX.