Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PYX: The next step in Python packaging

astral.sh

Post reply on HN
Print view View original

131–140 of 479 posts

Re: PYX: The next step in Python packaging

#131

Post by NeutralForest »

setopt wrote: post #85
morkalork wrote: post #61

Earlier quoted context omitted.

This was basically the reason to use anaconda back in the day.

In my experience, Anaconda (including Miniconda, Micromamba, IntelPython, et al.) is still the default choice in scientific computing and machine learning.

It's useful because it also packages a lot of other deps like CUDA drivers, DB drivers, git, openssl, etc. When you don't have admin rights, it's really handy to be able to install them and there's no other equivalent in the Python world. That being said, the fact conda (and derivatives) do not follow any of the PEPs about package management is driving me insane. The ergonomics are bad as well with defaults like auto activation of the base env and bad dependency solver for the longest time (fixed now), weird linking of shared libs, etc.

Re: PYX: The next step in Python packaging

#132

Post by no_wizard »

Good on you guys!

I wanted to start a business exactly like this years ago, when I actually worked in Python. I ended up not doing so, because at the time (circa 2014-2015) I was told it would never take off, no way to get funding.

I'm glad you're able to do what ultimately I was not!

Re: PYX: The next step in Python packaging

#133

Post by miohtama »

m_kos wrote: post #40

> Why is it so hard to install PyTorch, or CUDA, or libraries like FlashAttention or DeepSpeed that build against PyTorch and CUDA? This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience.

In the past, part of the definition of an operating system was that it ships with a compiler.

Re: PYX: The next step in Python packaging

#134

Post by scarlehoff »

charliermarsh wrote: post #54
m4r71n wrote: post #36

What does GPU-aware mean in terms of a registry? Will `uv` inspect my local GPU spec and decide what the best set of packages would be to pull from Pyx? Since this is a private, paid-for registry aimed at corporate clients, will there be an option to expose those registries externally as a public instance, but paid for by the company? That is, can I as a vendor pay for a Pyx registry for my own set of packages, and t…

> Will `uv` inspect my local GPU spec and decide what the best set of packages would be to pull from Pyx? We actually support this basic idea today, even without pyx. You can run (e.g.) `uv pip install --torch-backend=auto torch` to automatically install a version of PyTorch based on your machine's GPU from the PyTorch index. pyx takes that idea and pushes it further. Instead of "just" supporting PyTorch, the registr…

Hi Charlie

what happens in a situation in which I might have access to a login node, from which I can install packages, but then the computing nodes don't have internet access. Can I define in some hardware.toml the target system and install there even if my local system is different?

To be more specific, I'd like to do `uv --dump-system hardware.toml` in the computing node and then in the login node (or my laptop for that matter) just do `uv install my-package --target-system hardware.toml` and get an environment I can just copy over.

Re: PYX: The next step in Python packaging

#135

Post by threeducks »

ilvez wrote: post #70
giancarlostoro wrote: post #45

Earlier quoted context omitted.

Stuff like that led me fully away from Ruby (due to Rails), which is a shame, I see videos of people chugging along with Ruby and loving it, and it looks like a fun language, but when the only way I can get a dev environment setup for Rails is using DigitalOcean droplets, I've lost all interest. It would always fail at compiling something for Rails. I would have loved to partake in the Rails hype back in 2012, but ov…

Do I get it right that this issue is within Windows? I've never heard of the issues you describe while working with Linux.. I've seen people struggle with MacOS a bit due to brew different versions of some library or the other, mostly self compiling Ruby.

There certainly are issues on Linux as well. The Detectron2 library alone has several hundred issues related to incorrect versions of something: https://github.com/facebookresearch/detectron2/issues

The mmdetection library (https://github.com/open-mmlab/mmdetection/issues) also has hundreds of version-related issues. Admittedly, that library has not seen any updates for over a year now, but it is sad that things just break and become basically unusable on modern Linux operating systems because NVIDIA can't stop breaking backwards and forwards compatibility for what is essentially just fancy matrix multiplication.

Re: PYX: The next step in Python packaging

#136

Post by no_wizard »

nodesocket wrote: post #77
fph wrote: post #56

I lost track of how many different ways to install a Python library there are at the moment.

Much better than the Node a handful of years back. Everybody used NPM, everybody switched to Yarn, everybody switched back to NPM.

Thats because npm finally decided to adopt enough features as time went on that it could be used in place of yarn and eventually, if they adopt enough of the features of pnpm, it will replace that too.

Though speaking as a long time developer in the ecosystem, switching between npm, yarn, and pnpm is fairly trivial in my experience. Especially after node_gyp went away

Re: PYX: The next step in Python packaging

#137

Post by zahlman »

actinium226 wrote: post #86
asa400 wrote: post #49

Earlier quoted context omitted.

As someone outside those communities, could you elaborate?

Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/…

> any combination of windows/mac/linux + amd/arm + glibc/musl + CPython/pypy (did I miss any?).

From a standards perspective, it is a combination of a Python version/implementation, a "platform" and an "ABI". (After all, the glibc/musl distinction doesn't make sense on Windows.)

Aside from CPython/pypy, the system recognizes IronPython (a C# implementation) and Jython (a Java implementation) under the version "tag"; of course these implementations may have their own independent versioning with only a rough correspondence to CPython releases.

The ABI tag largely corresponds to the implementation and version tag, but for example for CPython builds it also indicates whether Python was built in debug or release mode, and from 3.13 onward whether the GIL is enabled.

The platform tag covers Mac, Windows, several generic glibc Linux standards (called "manylinux" and designed to smooth over minor differences between distros), and now also some generic musl Linux standards (called "musllinux"). Basic CPU information (arm vs intel, 32- vs 64-bit etc.) is also jammed in here.

Details are available at https://packaging.python.org/en/latest/specifications/platfo... .

Re: PYX: The next step in Python packaging

#139

Post by mnazzaro »

monster_truck wrote: post #10

I've been burned too many times by embracing open source products like this. We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place.

This is a valid concern, but astral just has an amazing track record.

I was surprised to see the community here on HN responding so cautiously. Been developing in python for about a decade now- whenever astral does something I get excited!

Re: PYX: The next step in Python packaging

#140

Post by happymellon »

woodruffw wrote: post #97
zahlman wrote: post #82

Earlier quoted context omitted.

> pyx uses the same PEP 503/691 interfaces as every other Python index ... Then how can it make decisions about how to serve the package request that PyPI can't? Is there not some extension to the protocol so that uv can tell it more about the client system?

The repository API allows server-driven content negotiation[1], so pyx can service specialized requests while also honoring the normal 503/691 ones. [1]: https://packaging.python.org/en/latest/specifications/simple...

Ah.

> honoring the normal 503/691 ones.

Embrace

> pyx can service specialized requests

Extend

... ;)

Post reply on HN