Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

151–160 of 204 posts

Re: How to improve Python packaging

#151

Post by pjmlp »

marsten wrote: post #99
samwillis wrote: post #13

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be…

> there will likely be a higher than average representation from people who would sit in the 10% And also an under-representation of "average users". Python the language has always benefitted from its relative simplicity, which I attribute to people like GVR saying "no" to specialized features that accrue on languages like barnacles on a ship (looking at you C++). With newer languages we see core design teams being m…

I know Python since version 1.6, its simplicity is deceptive, the language is as rich as C++, beginners only think otherwise because the language looks simple on the surface.

When one starts exploring the standard library, everything that is exposed in meta-programming and runtime internals, or the usual language changes even across minor versions, the picture changes dramatically.

Once upon a time I always devoured the release notes for each release, eventually I lost track of changes.

Re: How to improve Python packaging

#152

Post by Tigress8780 »

gen220 wrote: post #111
phailhaus wrote: post #52

Earlier quoted context omitted.

I'd say that Node's package management has run laps around Python's, to the point where it's pretty embarrassing. Since it's relatively new, it was able to hit the ground running with best practices: 1. Declarative package manifests. Python's ecosystem is still a mess of various approaches, and the fact that you have to run the setup.py script to determine dependencies is a nightmare. Because of this, running depende…

Respectfully, this reads like you're using outdated Python tools. Give Poetry [1] a shot, it has all the things you've listed here. Just as Node.js has come a long way in the last 5 years, Python has, too. Albeit, in a fashion that was much less centralized, arguably to Python's detriment. [1]: https://python-poetry.org/history/#100---2019-12-12 , released 1.0 in 2019/12.

Poetry can not give you things that do not (always) exist in the ecosystem, like properly declared dependencies [1], which are a must in most other package managers.

Poetry can be painfully slow if you happen to depend on many packages that only use `setup.py` to define dependencies.

[1] https://python-poetry.org/docs/faq/#why-is-the-dependency-re...

Re: How to improve Python packaging

#153

Post by rektide »

rektide wrote: post #47
Kwpolska wrote: post #36

Earlier quoted context omitted.

There’s a place for small packages, but is-even/is-odd is a bit too small to be a reasonable package. It is far easier to just write `x % 2 === 0` inline, which is an obvious idiom, instead of installing and importing a separate package for this. The use of is-odd by is-even can be confusing for users. For example, you may call isEven(0.5) and get the following error: RangeError: is-odd expects an integer. at isOdd (…

isEven is in thay stack trace - should not confuse anyone with even a basic introductory level fluency at coding. Is it too small? What if latter the language evolves BigInt? Donwe suffer a patchwork of libraries which have & havent upgraded, sussing around each time to find out? I think the key thing to recognize is that this is all opinion. Many people dont like the availability of many opions, the ease at which de…

Doubling down, there's just huge conceit that small modules are bad. And no evidence. People just love to hate. Dark side is easy & convenient & in reach, and one gains social clout, feels elite, by shitting on others. Boo to you.

It's not easy, it's not tame, but it's unclear what negativity beyond mild inconvenience has been generated. And much of the harm can be diffused with more sensible protection, not simply giving all modules access to everything. Systems like WASI are finally engineering inbuilt protection to sensibly de-risk imports; this is a fault of runtimes for not offering us protection, not our burgeoning package ecosystems for having value, growing.

It's still unclear what the worthwhile protests are.

Re: How to improve Python packaging

#154

Post by rpep »

My view is that it is not that hard (with any tool) to build a wheel. You can basically follow a guide, you have to supply the same information, it’s mostly just the format you do it in.

The problem is that compiled extensions are not a minor use case for Python, they’re hugely widely used for packages even if users don’t use them that much. Building a package with no dependencies is still not that hard. But there is no good way to distribute the compiled packages you depend on as their own thing - Intel release MKL as a “Python” package for e.g. but it is just the C library [1]. But of course, if you’re a package maintainer, the chance that the package maintainer of your dependency has both uploaded it to a package manager in a language they don’t use and that also being the version you need is pretty slim. So the issue becomes “how do I bundle every single possible dependency I might need inside a Wheel”, because source installs are not something most Python users understand. Conda and Spack try and fix this issue by being general package managers that can distribute packages in any language, allowing you to depend on FFTW or Eigen or SUNDIALS or whatever you need in your C extensions, and in all of the discussion I don’t think any proposal really tackles this gap.

[1] https://pypi.org/project/mkl/

Re: How to improve Python packaging

#155

Post by rtpg »

wheelerof4te wrote: post #70
matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.

to be fair to Python, almost every ecosystem is in "use Docker" land, even Go binaries will offer docker images!

Re: How to improve Python packaging

#156

Post by matsemann »

goodpoint wrote: post #65
matsemann wrote: post #7

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly clon…

No way. Docker is not designed for security.

Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...

Re: How to improve Python packaging

#157

Post by TigerTeamX »

theacodes wrote: post #140
TigerTeamX wrote: post #114

I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.

Please do elaborate before digging my own grave, thanks :D

Re: How to improve Python packaging

#158

Post by TigerTeamX »

thinker5555 wrote: post #136
TigerTeamX wrote: post #114

I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

I tinkered with this a little, and it's interesting. But what about using pip list or pip freeze? They don't show the packages in the .lib directory. Do you just add them by hand to the requirements.txt?

I add them by hand to requirements.txt and I try to keep it as minimum as possible. Yes it makes it not 100% reproducible, but it forces me to think about each dependency and I can see in my git log why it was added.

Re: How to improve Python packaging

#159

Post by wlpu »

optimalsolver wrote: post #8

Python packaging is a solved problem: https://python-poetry.org/

Poetry is one of the worst modern solutions to python packaging, either pick hatch or PDM, they both have more and better implemented features plus their codebases aren't a complete mess

Re: How to improve Python packaging

#160

Post by thinker5555 »

TigerTeamX wrote: post #158
thinker5555 wrote: post #136

Earlier quoted context omitted.

I tinkered with this a little, and it's interesting. But what about using pip list or pip freeze? They don't show the packages in the .lib directory. Do you just add them by hand to the requirements.txt?

I add them by hand to requirements.txt and I try to keep it as minimum as possible. Yes it makes it not 100% reproducible, but it forces me to think about each dependency and I can see in my git log why it was added.

It's funny how "pip help list" or "pip help freeze" works. :-) I found I can add --path .lib to list/freeze those. It's an extra step, but not too terrible.
Post reply on HN