Pypi.org is running a survey on the state of Python packaging • Hacker News Board

Re: Pypi.org is running a survey on the state of Python packaging

#91

Post by korijn » Wed, Sep 07, 2022, 8:59 PM UTC

dalke wrote: post #57

orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for Open…

I emphatize with your situation and it's a great example. As crazy as this may sound, I think you would have to build every possible permutation of your library and make all of them available on pypi. You'd need a some new mechanism based on metadata to represent all the options and figure out how to resolve against available system libraries. Especially that last part seems very complicated. But I do think it's possible.

Re: Pypi.org is running a survey on the state of Python packaging

#92

Post by orf » Wed, Sep 07, 2022, 9:01 PM UTC

korijn wrote: post #86

orf wrote: post #49

Earlier quoted context omitted.

Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling

Yeah. Well, mandating wheels and getting rid of setup.py at least avoids having to run scripts, and indeed enables the next step which would be indexing all the metadata and exposing it through an API. I just thought it wouldn't necessarily be obvious to all readers of your comment.

Just to be clear, package metadata already is sort of available through the pypi json api. I've got the entire set of all package metadata here: https://github.com/orf/pypi-data

  $ gzcat release_data/c/d/cdklabs.cdk-hyperledger-fabric-network.json.gz | jq '. | to_entries | .[].value.info.requires_dist' | head
  [
    "typeguard (~=2.13.3)",
    "publication (>=0.0.3)",
    "jsii (=1.63.2)",
    "constructs (=10.0.5)",
    "aws-cdk-lib (=2.33.0)"
  ]

It's just not everything has it, and there isn't a way to differenciate between "missing" and "no dependencies". And it's also only for the `dist` releases. But anyway, poetry uses this information during dependency resolution.

Re: Pypi.org is running a survey on the state of Python packaging

#93

Post by orf » Wed, Sep 07, 2022, 9:02 PM UTC

coldtea wrote: post #80

orf wrote: post #79

Earlier quoted context omitted.

It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.

You'd be surprised at how many billions lines of production code are still at 2 (and could not care less whether it's end-of-lined)

I'm not surprised at all, but regardless they also should not be similarly surprised if people could not care less about that use case.

Re: Pypi.org is running a survey on the state of Python packaging

#94

Post by mcdermott » Wed, Sep 07, 2022, 9:03 PM UTC

Installing packages, creating a manifest of dependancies, managing virtual environments, packaging, checking/formatting code, etc... should be built into the Python toolchain (the python binary itself). Needing to chose a bunch of third party tools to make it work... makes Python, well... un-pythonic.

Re: Pypi.org is running a survey on the state of Python packaging

#95

Post by at_a_remove » Wed, Sep 07, 2022, 9:13 PM UTC

qayxc wrote: post #87

at_a_remove wrote: post #11

I have a terrible admission to make: one of the reason I like Python is its huge standard library, and I like that because I just ... despise looking for libraries, trying to install them, evaluating their fitness, and so on. I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in…

> I view dependencies outside of the standard library as a kind of technical debt That's an interesting position. So are you suggesting that very specialised packages such as graph plotting, ML-packages, file formats, and image processing should be part of the standard library? What about very OS/hardware-specific packages, such as libraries for microcontrollers? There are many areas that don't have a common agreed-u…

Hrm. Graph-plotting ... yes. File formats ... yes, as many as possible. Image processing, given the success of ImageMagick, I'd say yes there as well. I don't know much about ML to say.

OS-specific packages, quite possibly.

The thing about the standard library is that it is like high school: there's a lot of stuff you think you will never need, and you're right about most of it, but the stuff you do need you're glad you had something going, at least.

Re: Pypi.org is running a survey on the state of Python packaging

#96

Post by at_a_remove » Wed, Sep 07, 2022, 9:15 PM UTC

rpcope1 wrote: post #54

mrweasel wrote: post #21

Earlier quoted context omitted.

If requests and a basic web framework was in the standard library you’d effectively eliminate the majority of my dependencies. Honestly I doubt see the package management being an issue for most end-users. Between the builtin venv, conda and Docker I feel that the use-cases for most is well covered. The only focus area I really see is better documentation. Easier to read documentation more precisely. Perhaps a set of…

Something like bottle.py would be an excellent candidate for inclusion. The real reason to avoid putting anything into the standard library is that it seems to often be the place where code goes to stagnate and die for Python.

I am not sure why that has turned into a truism.

Really good code in the standard library should reach a level of near perfection, then eventually transition into hopeful speed gains, after which you're really only changing that code because the language has changed or the specification has updated.

Re: Pypi.org is running a survey on the state of Python packaging

#97

Post by dalke » Wed, Sep 07, 2022, 9:15 PM UTC

orf wrote: post #81

dalke wrote: post #57

Earlier quoted context omitted.

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for Open…

To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or opti…

I agree it would be cool and useful.

But it appears to be such a hard problem that modern packaging tools ignore it, preferring to take on other challenges instead.

My own attempts at extracting Python configuration information to generate a Makefile for personal use (because Makefile understand dependencies better than setup.py) is a mess caused by my failure to understand what all the configuration options do.

Given that's the case, when do you think we'll be able to "Remove setup.py files and mandate wheels"?

I'm curious on what evils you're thinking of? I assume the need to run arbitrary Python code just to find metadata is one of them. But can't that be resolved with a pyproject.toml which uses setuptools only for the build backend? So you don't need to remove setup.py, only restrict when it's used, yes?

Re: Pypi.org is running a survey on the state of Python packaging

#98

Post by [deleted] » Wed, Sep 07, 2022, 9:16 PM UTC

ziml77 wrote: post #52

orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.

[deleted]

Re: Pypi.org is running a survey on the state of Python packaging

#99

Post by cycomanic » Wed, Sep 07, 2022, 9:49 PM UTC

ziml77 wrote: post #52

orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.

I know this is unpopular opinion on here, but I believe all this packaging madness is forced on us by languages because Windows (and to a lesser degree osx) have essentially no package management.

Especially installing a tool chain to compile C code for python is no issue on Linux, but such a pain on Windows.

Re: Pypi.org is running a survey on the state of Python packaging

#100

Post by dec0dedab0de » Wed, Sep 07, 2022, 9:57 PM UTC

orf wrote: post #81

dalke wrote: post #57

Earlier quoted context omitted.

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for Open…

To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or opti…

The ability to create a custom package that can run any custom code you want at install time is very powerful. I think a decent solution would be to have a way to mark a package as trusted, and only allow pre/post scripts if they are indeed trusted. Maybe even have specific permissions that can be granted, but that seems like a ton of work to get right across operating systems.

My specific use cases are adding custom CA certs to certifi after it is installed, and modifying the maximum version of a requirement listed for an abandoned library that works fine with a newer version.

I think the best solutions would be an official way to ignore dependencies for a specific package, and specify replacement packages in a project's dependencies. Something like this if it were a Pipfile:

  public-package = {version = "~=1.0",replace_with='path/to/local-package'}
  abandoned-package = {version = "~=*",ignore_dependencies=True}

But the specific problem doesn't matter, what matters is that there will always be exceptions. This is Python, we're all adults here, and we should be able to easily modify things to get them to work the way we want them to. Any protections added should include a way to be dangerous.

I know your point is more about requiring static metadata than using wheels per se. I just believe that all things Python should be flexible and hack-able. There are other more rigid languages if you're into that sort of thing.

edit:

before anyone starts getting angry I know there are other ways to solve the problems I mentioned.

forking/vendoring is a bit of overkill for such a small change, and doesn't solve for when a dependency of a dependency needs to be modified.

monkeypatching works fine, however it would need to be done at all the launch points of the project, and even then if I open a repl and import a specific module to try something it won't have my modifications.

modifying an installed package at runtime works reasonably well, but it can cause a performance hit at launch, and while it only needs to be run once, it still needs to be run once. So if the first thing you do after recreating a virualenv is to try something with an existing module we have the same problem as monkey patching.

'just use docker' or maybe the more toned down version: 'create a real setup script for developers' are both valid solutions, and where I'll probably end up. It was just very useful to be able to modify things in a pinch.