Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

Post reply on HN
Print view View original

101–110 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#101

Post by IceHegel »

If I see some JS, Go, or Rust code online I know I can probably get it running on my machine in less than 5 min. Most of the time, it's a ‘git clone’ and a 'yarn' | 'go install' | 'cargo run', and it just works.

With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often very opaque errors. (The last 2 years on M1 were really rough)

Setting up Pytorch or Tensorflow + CUDA is a nightmare I've experienced many times.

Having so many ways to manage packages is especially harmful for python because many of those writing python are not professional software engineers, but academics and researchers. If they write something that needs, for example, CUDA 10.2, Python 3.6, and a bunch of C audio drivers - good luck getting that code to work in less than a week. They aren’t writing install scripts, or testing their code on different platforms, and the python ecosystem makes the whole process worse by providing 15 ways of doing basically the same thing.

My proposal:

- Make poetry part of pip

- Make local installation the default (must pass -g for global)

- Provide official first party tooling for starting a new package

- Provide official first party tooling for migrating old dependency setups to the new standard

edit: fmt

Re: Pypi.org is running a survey on the state of Python packaging

#102

Post by IceHegel »

shrimpx wrote: post #38

I like poetry for its simplicity but I can’t tell how “official” it is in the python ecosystem. I hope it doesn’t die out. I think it’s the simplest possible way to maintain deps and publish to PyPI if you don’t have any weird edge cases.

It should be make the official package manager IMO.

Re: Pypi.org is running a survey on the state of Python packaging

#103

Post by infogulch »

dalke wrote: post #97
orf wrote: post #81

Earlier quoted context omitted.

To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or opti…

I agree it would be cool and useful. But it appears to be such a hard problem that modern packaging tools ignore it, preferring to take on other challenges instead. My own attempts at extracting Python configuration information to generate a Makefile for personal use (because Makefile understand dependencies better than setup.py) is a mess caused by my failure to understand what all the configuration options do. Give…

The closest thing I've seen to a solution in this space is Riff, discussed yesterday [1], which solves the external dependency problem for rust projects.

[1]: https://news.ycombinator.com/item?id=32739954

Re: Pypi.org is running a survey on the state of Python packaging

#104

Post by urschrei »

progval wrote: post #70
orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

> For improvements I commented: Remove setup.py files and mandate wheels. This would make most C extensions impossible to install on anything other than x86_64-pc-linux-gnu (or arm-linux-gnueabihf/aarch64-linux-gnu if you are lucky) because developers don't want to bother building wheels for them.

cibuildwheel (which is an official, supported tool) has made this enormously easier. I test and generate wheels with a compiled (Rust! Because of course) extension using a Cython bridge for all supported Python versions for 32-bit and 64-bit Windows, macOS x86_64 and arm64, and whatever manylinux is calling itself this week. No user compilation required. It took about half a day to set up, and is extremely well documented.

Re: Pypi.org is running a survey on the state of Python packaging

#105

Post by bno1 »

IceHegel wrote: post #101

If I see some JS, Go, or Rust code online I know I can probably get it running on my machine in less than 5 min. Most of the time, it's a ‘git clone’ and a 'yarn' | 'go install' | 'cargo run', and it just works. With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often ver…

I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.

Re: Pypi.org is running a survey on the state of Python packaging

#106

Post by qayxc »

at_a_remove wrote: post #95
qayxc wrote: post #87

Earlier quoted context omitted.

> I view dependencies outside of the standard library as a kind of technical debt That's an interesting position. So are you suggesting that very specialised packages such as graph plotting, ML-packages, file formats, and image processing should be part of the standard library? What about very OS/hardware-specific packages, such as libraries for microcontrollers? There are many areas that don't have a common agreed-u…

Hrm. Graph-plotting ... yes. File formats ... yes, as many as possible. Image processing, given the success of ImageMagick, I'd say yes there as well. I don't know much about ML to say. OS-specific packages, quite possibly. The thing about the standard library is that it is like high school: there's a lot of stuff you think you will never need, and you're right about most of it, but the stuff you do need you're glad…

ImageMagick is actually a good example: I use Python as my primary tool for shell scripting (I don't like "traditional" shell scripts for various reasons) - if I can use Python to control external tools such as ImageMagick, why would I want to include all its functionality, codecs, effects, etc. in the standard library?

Including too much leads to a huge burden for the maintainers and consequently results in this: https://peps.python.org/pep-0594/

Quote:

> Times have changed. With the introduction of PyPI (née Cheeseshop), setuptools, and later pip, it became simple and straightforward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third-party packages. It’s pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.

> On the other hand, Python’s standard library is piling up with cruft, unnecessary duplication of functionality, and dispensable features.

Re: Pypi.org is running a survey on the state of Python packaging

#107

Post by humanrebar »

cycomanic wrote: post #99
ziml77 wrote: post #52

Earlier quoted context omitted.

I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.

I know this is unpopular opinion on here, but I believe all this packaging madness is forced on us by languages because Windows (and to a lesser degree osx) have essentially no package management. Especially installing a tool chain to compile C code for python is no issue on Linux, but such a pain on Windows.

C tends to work in those cases because there aren't a significant number of interesting C dependencies to add... because there is no standard C build system, packaging format, or packaging tools.

When juggling as many transitive dependencies in C as folks do with node, python, etc., there's plenty of pain to deal with.

Re: Pypi.org is running a survey on the state of Python packaging

#108

Post by KingEllis »

crazytalk wrote: post #88
clintonb wrote: post #3

The survey is at https://www.surveymonkey.co.uk/r/M5XKQCT .

This survey is the literal definition of leading question. Found about 2 boxes I could tick, before being forced to order a list of the designer's preferences according to how much I agree with them. The only data that can be generated from a survey like this is the data you wanted to find (see also Boston Consulting Group article earlier today). I cannot honestly respond to it The only question I have is, what grant…

The absence of the go binary as a tool (i.e. "go get ...", "go install ..." etc.) is odd, considering that is what has been eating Python's lunch lately.

Re: Pypi.org is running a survey on the state of Python packaging

#109

Post by antod »

shireboy wrote: post #30

I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python. It looks to me like people are trying, but different libraries use different techniques. To a newcomer this is confusing.

>I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python.

Python has had multiple legacy solutions going back a long time before nuget and npm existed, and before central registries of dependencies. Every new solution has to cope with all that compatibility/transitional baggage. Also a bunch of usecases .NET or JS never really had to deal much with - eg being a core system language for Linux distros, and supporting cross platform installs back in the download and run something days. The scope of areas Python gets used in means its packaging is pulled in more directions than most other languages who mostly stick to a main niche.

So the history and surface area of problems to solve in Python packaging is larger than what most other languages have had to deal with. It also takes years for the many 3rd party tools to try out new approaches, gain traction and then slowly get their best ideas synthesized and adapted into the much more conservative core Python stdlib.

Not saying it is great, just laying out some of the reasons it is what it is.

Re: Pypi.org is running a survey on the state of Python packaging

#110

Post by orf »

mistrial9 wrote: post #90
orf wrote: post #79

Earlier quoted context omitted.

It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.

very unfortunately the direct burden of python2 is placed on the packagers.. users of Python 2 like their libs (me) and have no horse in this demonization campaign

Pay for support for Python 2 then? At which point it’s a burden on the person you are paying. Or don't, in which case you're complaining that people are demonizing you because they are not doing your work for free?
Post reply on HN