Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

Post reply on HN
Print view View original

81–90 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#81

Post by orf »

dalke wrote: post #57
orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for Open…

To be clear, I’m not suggesting we remove the ability to compile native extensions.

I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py.

It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way.

Think of it like cargo.toml va build.rs.

Re: Pypi.org is running a survey on the state of Python packaging

#82

Post by shrimpx »

tempest_ wrote: post #65
wlkr wrote: post #56

Earlier quoted context omitted.

Couldn't agree more. Poetry is fantastic and provides that 'just works' experience for most cases. It's not official (although possibly should be adopted) but has gained ground by virtue of its quality. Fortunately it's very actively developed so will hopefully stick around.

I like poetry. It still has a way to go though since it is slow as all hell doing almost anything and its error messages are closer to a stack trace than something actionable.

I agree that the stack trace error messages are weird. That aspect feels uncharacteristically hacky for an otherwise pretty polished tool.

Re: Pypi.org is running a survey on the state of Python packaging

#83

Post by dalke »

mistrial9 wrote: post #68
dalke wrote: post #58

Earlier quoted context omitted.

What if I have a dependency on a commercial third-party Python package which is on Conda but not on PyPI?

you are placing open code in a vendor lock-in, to start

Yes, I understand that.

I see I misunderstood korijn's comment. My earlier reply is off-topic, so I won't continue further off the track.

Re: Pypi.org is running a survey on the state of Python packaging

#84

Post by dalke »

ciupicri wrote: post #74
dalke wrote: post #64

Earlier quoted context omitted.

Should PyPI kick my project off because it don't support MS Windows? My package uses C and Cython extensions. While I support macOS and Linux-based OSes, I don't know how to develop on or support MS Windows. I've tried to be careful about the ILP32 vs LP64 differences, but I suspect there's going to be many places where I missed up. I also use "/dev/stdin" to work-around my use of a third-party library that has no wa…

Can't you use "CON" instead of "/dev/stdin" on Windows?

I asked that question nearly 11 years ago on StackOverflow, at https://stackoverflow.com/questions/7395157/windows-equivale... . ;)

Quoting the best comment, "echo test | type CON or echo test | type CONIN$ will read from the console, not from stdin."

Re: Pypi.org is running a survey on the state of Python packaging

#85

Post by tux3 »

ziml77 wrote: post #52
orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.

It feels so suboptimal to need the C toolchain to do things, but having no solid way to depend on it as a non-C library (especially annoying in Rust, which insists on building everything from source and never installing libraries globally).

I make a tool/library that requires the C toolchain at runtime. That's even worse than build time, I need end users to have things like lld, objdump, ranlib, etc installed anywhere they use it. My options are essentially:

- Requiring users to just figure it out with their system package manager

- Building the C toolchain from source at build time and statically linking it (so you get to spend an hour or two recompiling all of LLVM each time you update or clear your package cache! Awesome!),

- Building just LLD/objdump/.. at build-time (but user still need to install LLVM. So you get both slow installs AND have to deal with finding a compatible copy of libLLVM),

- Pre-compiling all the C tools and putting them in a storage bucket somewhere, for all architectures and all OS versions. But then not have support when things like the M1 or new OS versions right away, or people on uncommon OSes. And now need to maintain a build machine for all of these myself.

- Pre-compile the whole C toolchain to WASM, build Wasmtime from source instead, and just eat the cost of Cranelift running LLVM 5-10x slower than natively...

I keep trying to work around the C toolchain, but I still can't see any very good solution that doesn't make my users have extra problems one way or another.

Hey RiiR evangelism people, anyone want to tackle all of LLVM? .. no? No one? :)

Re: Pypi.org is running a survey on the state of Python packaging

#86

Post by korijn »

orf wrote: post #49
korijn wrote: post #48

Earlier quoted context omitted.

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling

Yeah. Well, mandating wheels and getting rid of setup.py at least avoids having to run scripts, and indeed enables the next step which would be indexing all the metadata and exposing it through an API. I just thought it wouldn't necessarily be obvious to all readers of your comment.

Re: Pypi.org is running a survey on the state of Python packaging

#87

Post by qayxc »

at_a_remove wrote: post #11

I have a terrible admission to make: one of the reason I like Python is its huge standard library, and I like that because I just ... despise looking for libraries, trying to install them, evaluating their fitness, and so on. I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in…

> I view dependencies outside of the standard library as a kind of technical debt

That's an interesting position. So are you suggesting that very specialised packages such as graph plotting, ML-packages, file formats, and image processing should be part of the standard library? What about very OS/hardware-specific packages, such as libraries for microcontrollers?

There are many areas that don't have a common agreed-upon set of idioms or functionality and that are way too specialised to be useful for most users. I really don't think putting those into the standard library would be a good idea.

Re: Pypi.org is running a survey on the state of Python packaging

#88

Post by crazytalk »

clintonb wrote: post #3

The survey is at https://www.surveymonkey.co.uk/r/M5XKQCT .

This survey is the literal definition of leading question. Found about 2 boxes I could tick, before being forced to order a list of the designer's preferences according to how much I agree with them. The only data that can be generated from a survey like this is the data you wanted to find (see also Boston Consulting Group article earlier today). I cannot honestly respond to it

The only question I have is, what grant application(s) is the survey data being used to support?

Re: Pypi.org is running a survey on the state of Python packaging

#89

Post by baggiponte »

orf wrote: post #41

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixin…

I recommend PDM over poetry!

Re: Pypi.org is running a survey on the state of Python packaging

#90

Post by mistrial9 »

orf wrote: post #79
kevin_thibedeau wrote: post #76

Earlier quoted context omitted.

Setup.py can do things wheels can't. Most notably it's the only installation method that can invoke 2to3 at runtime without requiring a dev to create multiple packages.

It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.

very unfortunately the direct burden of python2 is placed on the packagers.. users of Python 2 like their libs (me) and have no horse in this demonization campaign
Post reply on HN