Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PYX: The next step in Python packaging

astral.sh

Post reply on HN
Print view View original

381–390 of 479 posts

Re: PYX: The next step in Python packaging

#381

Post by physicsguy »

cyberax wrote: post #166
actinium226 wrote: post #86

Earlier quoted context omitted.

Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/…

> Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. And in the TS/JS world we have React.Native that has a flexible pluggable model that allows creating XCode projects with autodiscovered dependencies in C, C++, Swift and other languages. It's also flexible enough to allow third-party products like Sentry to integrate into the build process to upload d…

Come on, React Native is a thing but as a % of JS engineers, the number that will regularly stray into compiled packages is much smaller than with Python. Basically every time you pick up a Maths or ML project which is a huge part of Python's ecosystem, you'll be using C or Fortran code underneath. That library ecosystem in that are is just non-existent in JS, and where it does people are typically using compile-to-WASM anyway.

Re: PYX: The next step in Python packaging

#382

Post by physicsguy »

semiquaver wrote: post #240
int_19h wrote: post #195

Earlier quoted context omitted.

Ruby is mostly used in web dev, where most if not all of your dependencies tend to be pure Ruby. Python is used heavily for DS/ML/AI, which is exactly the area where native code packages are necessary and prevalent. Worse yet is that those packages often involve GPU code, and things like CUDA bring their own complications. If you're writing web apps in Python, dependencies haven't really been a problem for a long tim…

> Ruby is mostly used in web dev, where most if not all of your dependencies tend to be pure Ruby. There is literally no such thing as a rails app that’s pure ruby. Rails depends on nokogiri, which is a libxml2 wrapper, and all activerecord database adapters are C bindings. Ruby development involves dealing with frequent native extension compilation, just like python.

Those are all fine and easy dependencies to install in Python too though, lxml comes in as part of the Python standard lib, and database adaptors like psycopg all bundle the full source code needed.

Where it breaks down in Python is when the dependencies are enormous source code projects with their own dependencies in their own right, that you can't reasonably bundle in with your own project. Conda aimed to solve that by being cross-platform. In the old Python days you'd do 'pip install X' and it would fail because it expected the package to be available, on your system, and in the PATH/LD_LIBRARY_PATH. Then wheels (python packages that could bundle libraries rather than just source code) came along and people started bundling lowest-common-denominator versions that most people could use but at non-optimal performance, and that's roughly still where we are today.

Re: PYX: The next step in Python packaging

#383

Post by aragilar »

woodruffw wrote: post #13
TheChaplain wrote: post #6

Pyx is just a registry, just like Pypi, or did I misunderstood it?

Not exactly -- part of pyx is a registry (and that part speaks the same standards as PyPI), but the bigger picture is that pyx part of a larger effort to make Python packaging faster and more cohesive for developers. To be precise: pyx isn't intended to be a public registry or a free service; it's something Astral will be selling. It'll support private packages and corporate use cases that are (reasonably IMO) beyond…

I'm confused how pyx would achieve this. In my experience unless uv/pyx is planning on pursuing being a new conda (and even then it's pretty incomplete), a single cohesive isn't possible because uv/pyx can't be the universal single source of truth. The value of Python has always been its ability to be integrated into a larger whole, and so the Python tooling must always defer to the tooling for the larger whole (and that space is composed of numerous competing tools and ecosystems). PyPI-provided non-pure wheels don't integrate well outside the ecosystem that's been built there, so I don't see how doubling down on it solves the underlying problem?

Re: PYX: The next step in Python packaging

#384

Post by physicsguy »

zahlman wrote: post #154
kristoff_it wrote: post #115

Is this going to solve the combinatorial explosion of pre-building native dependencies for every possible target? Python should get rid of its training wheels :^) https://kristoff.it/blog/python-training-wheels/

> Emphasis mine. It would indeed be hard to survive without that kind of support from a corporation. A user on HN estimated the yearly cost of this traffic at around 12 million USD/year (according to AWS Cloudfront rates), more than four times the full operating budget of the Python Software Foundation as of 2024. (As the user in question: three times.) > Leverage progress in the systems programming ecosystem to crea…

> Windows users are just not going to be able to build that locally.

I think that's being too kind to Linux + Mac users, they are often not generally able to do it either in my experience, especially with Python stuff which tends to need more bleeding edge deps than the local package manager can supply.

Re: PYX: The next step in Python packaging

#385

Post by microtonal »

imtringued wrote: post #371
hulitu wrote: post #282

Earlier quoted context omitted.

Your comment shows the sad state of software quality those days. Rust is the same, move fast and break things. And lately also Mesa started to suffer from the same disease. You basically need, those days, the same build env like the one on the developer's machine or the build will fail.

Bad software quality is when you update your software frequently. Instead, we should always choose the common denominator of the most obsolete software platform imaginable. If there is an OS that has not been maintained for several decades, then that is the baseline we should strive to support. Using an operating system with old libraries and language runtimes is not a personal preference with the consequences of res…

Please no, I have to deal with old (but still supported) RHEL versions, this is definitely not the way to go.

You have to use ancient C++ standard versions, deal with bugs in libraries that have been fixed years ago, lose out on all kinds of useful improvements or you end up with retrofitting a modern toolchain on an old system (but you still have to deal with an old glibc).

It’s hell. Just make the tooling/QA good enough so that everyone can run on the latest stable OS not too long after it’s released.

Re: PYX: The next step in Python packaging

#386

Post by JimDabell »

aragilar wrote: post #368
JimDabell wrote: post #328

Earlier quoted context omitted.

Frankly, it’s weird . You can find this business model all over the open-source world but for some reason Astral in particular is singled out for way more criticism on this than anything else I’ve seen, despite being unambiguously great contributors who have never put a foot wrong as far as I can tell. Microsoft – who invented embrace, extend, and extinguish – own NPM, but I don’t see people wringing their hands over…

NPM has always been commercial (rather than managed by a foundation), and it was nominally acquired by GitHub rather than Microsoft, so at some level as long as GitHub is not causing issues (noting the recent GitHub changes should maybe also imply some consideration of problems for NPM), NPM is "safe". Astral on the other hand has basically been rewrites in Rust of existing community-based open source tools, for whic…

> as long as GitHub is not causing issues

Astral are not causing issues though. Why does “as long as Astral is not causing issues” not apply?

> Anaconda/ContinuumIO was also treated with similar suspicion to Astral

I haven’t observed this. I have seen condo talked about a fair amount but any issues have always revolved around it being awkward to use. But practically every discussion here or on Reddit about Astral has FUD.

Re: PYX: The next step in Python packaging

#387

Post by procaryote »

AlphaSite wrote: post #90
dakiol wrote: post #39

I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…

I appreciate everything they’ve done but the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity. It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem. I suspect it’s also a misalignment of interests. No one there really invests in improving UX.

Last I checked they couldn't even figure out rate limiting or mirroring to make `pip search` work from the commandline without the index falling over

The bits of it that work are useful though

Re: PYX: The next step in Python packaging

#388

Post by codethief »

alexshagov wrote: post #357

I personally use Nixpkgs to fully isolate python installation on per project basis. And inside this isolated env, once python is built, usually using pip is enough and works well This way, !each of my repositories has its own nix file defining the list of dependencies and it can easily be built per system as well

I love Nix as much as the next guy, but I also need my coworkers to be able to get things to work on their non-NixOS machines. Have you found an easy way to have Nix install packages through uv (from a pyproject.toml / uv.lock files)? Right now I just do programs.nix-ld.enable = true; and use uv completely separately from Nix.

Re: PYX: The next step in Python packaging

#389

Post by procaryote »

dakiol wrote: post #39

I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…

Unless the thing you're building is a python library you want other people to install, you don't need to understand all that much of it.

You just need to understand how to setup a venv, and for each thing you want to depend on how to install it in the venv. Put those bits in a shell script and you can clone the project, run the script and then have a working env

It will break sometimes because python and a lot of python packages have zero respect for backward compatibility, but that's python for you.

It's ugly and horrible, but you don't have to relearn the latest python packaging "standard" every other year

Re: PYX: The next step in Python packaging

#390

Post by anentropic »

rcleveng wrote: post #300

Just try using ux with Google Artifact Registry and AWS CodeArtifact. You need the username in the URL with GAR and you need it in the environment variable with AWS. I'm not sure who's causing the pain with using uv with private repositories (but I'd bet it's uv), but they are selling a solution to that pain. Full disclosure, uv rocks and is way better than poetry, setuptools and whatever complicated and non-zen way…

any kind of AWS authentication is always a pain in the neck
Post reply on HN