Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PYX: The next step in Python packaging

astral.sh

Post reply on HN
Print view View original

391–400 of 479 posts

Re: PYX: The next step in Python packaging

#391

Post by procaryote »

nhumrich wrote: post #181
nixpulvis wrote: post #169

Can I ask a dumb question. Why does Ruby (for example) not have this problem, but python still can't ship a standard solution which isn't constantly changing and rolled up in some corporate offering?

GPU/c-bindings. Python packaging for for Python only modules has never been a problem. When people say they hate python packaging they are usually talking about being able to successfully install dependencies without much thinking. But, the biggest reason that doesn't work is because of the dependencies that have to be compiled. Which brings it's own problems. Have you ever had a c dependency on node or ruby on a sys…

Python packaging for pure python has definitely been a problem with competing methods and limitations.

Venv hasn't always been around for example, and before it was, dependency conflicts between two things running on the same machine were hairy

Re: PYX: The next step in Python packaging

#392

Post by demurgos »

actinium226 wrote: post #86
asa400 wrote: post #49

Earlier quoted context omitted.

As someone outside those communities, could you elaborate?

Not the person you're replying to, so I don't know if this is what he had in mind, but with Python packages you can distribute more than just Python. Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. Of course you can't expect everyone to have a dev environment set up, so packages can also contain pre-compiled binary for any combination of windows/mac/…

> I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS?

As a sibling comment posted, Node packages have no issue distributing non-JS assets. Since the very beginning, the "package.json" file had support for the "arch" and "postinstall" fields to distribute precompiled binaries or compile on install, for example using node-gyp. If you split your precompiled binaries and use optional dependencies then only the right package for your machine is downloaded.

In all package managers, distributing native code may be uncommon but it's always supported well enough (Node/npm/yarn, JVM/Gradle, PHP/Composer, Ruby/gem, Rust/Cargo, etc). What's unique about Python is how messy all the packaging handling is compared to other languages.

Re: PYX: The next step in Python packaging

#393

Post by hexo »

rmonvfer wrote: post #356

To be honest, this was just a matter of time. As a long time Python developer, I just can’t wrap my head around the lack of something like this. GitHub was going to get hosted packages for Python but never did because it “didn’t align with their strategy objectives and a reallocation of resources” [1] (or some other similar corpospeak) Astral is a great company and I think we can’t question what they’ve achieved and…

to be honest. ill never use uv. python ecosystem tools should be in python.

Re: PYX: The next step in Python packaging

#394

Post by btreecat »

bastawhiz wrote: post #204
runningmike wrote: post #103

All python packaging challenges are solved. Lesson learned is that there is not a single solution for all problems. getting more strings attached with VC funded companies and leaning on their infrastructure is a high risk for any FOSS community.

Well I started with pip because it's what I was told to use. But it was slow and had footguns. And then I started using virtualenv, but that only solved part of the problem. So I switched to conda, which sometimes worked but wrecked my shell profile and often leads to things mysteriously using the wrong version of a package. So someone told me to use pipenv, which was great until it was abandoned and picked up by som…

> But the dependency I need is built and packaged differently for different operating systems and flavor of GPU, and now my coworkers can't get the project to install on their laptops.

This is why containers are great IMO.

It's meant to solve the problem of "well it works on my machine"

Re: PYX: The next step in Python packaging

#395

Post by demurgos »

physicsguy wrote: post #381
cyberax wrote: post #166

Earlier quoted context omitted.

> Some packages contain C/C++/Fortran/Rust/others? source code that pip will try to automatically build upon install. And in the TS/JS world we have React.Native that has a flexible pluggable model that allows creating XCode projects with autodiscovered dependencies in C, C++, Swift and other languages. It's also flexible enough to allow third-party products like Sentry to integrate into the build process to upload d…

Come on, React Native is a thing but as a % of JS engineers, the number that will regularly stray into compiled packages is much smaller than with Python. Basically every time you pick up a Maths or ML project which is a huge part of Python's ecosystem, you'll be using C or Fortran code underneath. That library ecosystem in that are is just non-existent in JS, and where it does people are typically using compile-to-W…

React native is just an example, the point is that the npm registry has no issue distributing binaries.

Sass, Prisma, native DB drivers, or any other project using node-gyp or Node's NAPI are valid examples.

Re: PYX: The next step in Python packaging

#396

Post by orhmeh09 »

avidphantasm wrote: post #160
giancarlostoro wrote: post #45

Earlier quoted context omitted.

Stuff like that led me fully away from Ruby (due to Rails), which is a shame, I see videos of people chugging along with Ruby and loving it, and it looks like a fun language, but when the only way I can get a dev environment setup for Rails is using DigitalOcean droplets, I've lost all interest. It would always fail at compiling something for Rails. I would have loved to partake in the Rails hype back in 2012, but ov…

Have you tried conda? Since the integration of mamba its solver is fast and the breadth of packages is impressive. Also, if you have to support Windows and Python with native extensions, conda is a godsend.

It is not fast. Mamba and micromamba are still much faster than conda and yet lack basic features that conda has to provide. Everyone is dropping conda like a hot plate since the licensing changes in 2024.

Re: PYX: The next step in Python packaging

#397

Post by orhmeh09 »

awesome_dude wrote: post #68
giancarlostoro wrote: post #45

Earlier quoted context omitted.

Stuff like that led me fully away from Ruby (due to Rails), which is a shame, I see videos of people chugging along with Ruby and loving it, and it looks like a fun language, but when the only way I can get a dev environment setup for Rails is using DigitalOcean droplets, I've lost all interest. It would always fail at compiling something for Rails. I would have loved to partake in the Rails hype back in 2012, but ov…

Lets be honest here - whilst some experiences are better/worse than others, there doesn't seem to be a dependency management system that isn't (at least half) broken. I use Go a lot, the journey has been - No dependency management - Glide - Depmod - I forget the name of the precursor - I just remembered, VGo - Modules We still have proxying, vendoring, versioning problems Python: VirtualEnv Rust: Cargo Java: Maven an…

I think CRAN for R is very good, partly aided by an aggressive pruning policy for broken packages.

Re: PYX: The next step in Python packaging

#398

Post by cma256 »

hexo wrote: post #393
rmonvfer wrote: post #356

To be honest, this was just a matter of time. As a long time Python developer, I just can’t wrap my head around the lack of something like this. GitHub was going to get hosted packages for Python but never did because it “didn’t align with their strategy objectives and a reallocation of resources” [1] (or some other similar corpospeak) Astral is a great company and I think we can’t question what they’ve achieved and…

to be honest. ill never use uv. python ecosystem tools should be in python.

Wait until you find out what language Python is written in.

Re: PYX: The next step in Python packaging

#399

Post by physicsguy »

demurgos wrote: post #395
physicsguy wrote: post #381

Earlier quoted context omitted.

Come on, React Native is a thing but as a % of JS engineers, the number that will regularly stray into compiled packages is much smaller than with Python. Basically every time you pick up a Maths or ML project which is a huge part of Python's ecosystem, you'll be using C or Fortran code underneath. That library ecosystem in that are is just non-existent in JS, and where it does people are typically using compile-to-W…

React native is just an example, the point is that the npm registry has no issue distributing binaries. Sass, Prisma, native DB drivers, or any other project using node-gyp or Node's NAPI are valid examples.

Pypi doesn't have any issue distributing binaries either these days? The issue is (a) diversity of CPU/GPU microarchitectures inc. CUDA (b) diversity of OS-es.

I've never had to compile a native Node library on a POWER9 GPU cluster, but I have had to compile Python ones.

Re: PYX: The next step in Python packaging

#400

Post by skydhash »

pletnes wrote: post #339
mavamaarten wrote: post #333

Earlier quoted context omitted.

The problem for me: a non-python developer, is that I just don't know what to do, ever, to run an existing script or program. It seems every project out there uses a different package manager, a different version of python, a different config file to set all of that up. Most of the time, I just have a random .py file somewhere. Sometimes it's a full project that I can look at and find out what package manager it's us…

I believe what is missing is a way of distributing apps. You face similar issues if you get the C++ source of a random program - there are quite a few build systems in use! However, the compiled program can often just be zipped and shipped, somehow.

The C/C++ ecosystem is a bit more sane, but requires more expertise to fix. As long as you figure out the build process, usually you can rely on the distro packages. For Node and Rust, people really like to use the latest version and not the LTS one for their software.
Post reply on HN