Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

151–160 of 174 posts

Re: Python's New Package Landscape

#151

Post by toyg »

qwerty456127 wrote: post #140
kankroc wrote: post #47

Earlier quoted context omitted.

Can you elaborate on why using a requirements.txt file is "manual hell"? I rely on it for pretty much everything and I didn't run into game breaking problems.

The only problem I know with requirements.txt is that many people would require particular versions there while later versions work perfectly fine. Every time I clone someone's Python project to work with I have to manually replace all the =s with >=s to avoid downloading obsolete versions of the dependencies and have never encountered a problem. Anyway, for me the most annoying thing about the Python projects archit…

> have never encountered a problem.

Oh, so you weren't around when Requests went 2.0 backward-incompatible (because they changed .json() with .json, or the other way around, can't remember) and half of PyPI, with its happy-go-lucky ">=1.0", broke...?

Since then, most people have learnt that you pin first and ask questions later.

Re: Python's New Package Landscape

#152

Post by luord »

I rarely even use requirements.txt and never use it in my personal projects.

I just pin the project's direct dependencies in the setup.py file and install the folder directly. I know it might cause bugs with different developers (or the CI) using different versions of the upstream dependencies but I guess I trust the developers who create each library I'm using. The moment I directly import something from what used to be an upstream dependency, I pin it too.

So far this approach hasn't given me trouble, but I'll still take a look at poetry based on what I read in the comments here.

Re: Python's New Package Landscape

#153

Post by andersriutta »

sirn wrote: post #65
rhizome31 wrote: post #17

Earlier quoted context omitted.

It seems that Anaconda is under-appreciated outside of pydata circles. Before using it I had no idea that it could manage virtual environments, dependencies and different versions of python. The fact that it's not a community-driven project might be one of the reasons. Meanwhile, in a galaxy far away, people are also using buildout.

I have been revisiting buildout recently and I wish there's something that merge ease of use of pipenv with buildout concept. Perhaps something similar to Nix throw in the mix, but more specific to a Python project. I heard that I can do this with Conda, but I never tried. Being able to define and install external dependencies (e.g. ImageMagick, libsodium, etc.) from a configuration file local to a project is somethi…

> Perhaps something similar to Nix throw in the mix, but more specific to a Python project.

Any examples of how Nix itself doesn't do what you need? One example I can think of: Nix doesn't support Windows.

https://nixos.org/nixpkgs/manual/#python

Re: Python's New Package Landscape

#154

Post by toyg »

Serow225 wrote: post #150
jonafato wrote: post #130

Earlier quoted context omitted.

The PyPA team has done a lot over the past five years. The changelog for pip ( https://pip.pypa.io/en/stable/news/ ) contains quite a bit, PyPI was migrated to Warehouse, and there have been several PEPs focused on improving the packaging situation. A lot of these ideas come from various people in the community and get formalized as official recommendations or tools, but these things take time, especially accounting…

Understood, I guess I'm wondering why it hasn't been possible to cull more of the less-successful attempts, or at least make it obvious to newer users what is legacy. As an outsider/newer person to Python, the number of package mgmt options to consider is vast and confusing, it would be helpful if there was one (or a few) more "blessed" solutions :)

> the number of package mgmt options to consider is vast and confusing

Part of the issue is due to the success of Python in very different niches. The likes of Rails or Node can concentrate on specific ecosystems, which account for the bulk of their users and have a limited set of scenarios they have to support; whereas Python users come from sysadmin to data-crunching to web to desktop development to games to to to...

So each packaging tool comes with certain ideas, usually a result of the author's experience; maybe they work very well in this or that scenario, but then they break badly on others and sizeable chunks of the community revolt. So a new tool comes around and the cycle starts again, but now people also want compatibility with the old tool.

I suspect part of the solution will require splits between niches. It already happened with Anaconda, which has basically become the standard in a particular group of users (academia / datascience). Since that came around, lamentations around building C libraries have substantially reduced (to be fair, the arrival of precompiled wheels on PyPI also helped). Some similarly-specialized tool might eventually emerge as standard for other niches.

Python developers are cats and they are pretty hard to herd at the best of times, which is unsurprising -- who would stick around a language that is almost 30 years old and was never promoted by any major vendor? Only hard-headed fools like myself.

Re: Python's New Package Landscape

#155

Post by qwerty456127 »

toyg wrote: post #151
qwerty456127 wrote: post #140

Earlier quoted context omitted.

The only problem I know with requirements.txt is that many people would require particular versions there while later versions work perfectly fine. Every time I clone someone's Python project to work with I have to manually replace all the =s with >=s to avoid downloading obsolete versions of the dependencies and have never encountered a problem. Anyway, for me the most annoying thing about the Python projects archit…

> have never encountered a problem. Oh, so you weren't around when Requests went 2.0 backward-incompatible (because they changed .json() with .json, or the other way around, can't remember) and half of PyPI, with its happy-go-lucky ">=1.0", broke...? Since then, most people have learnt that you pin first and ask questions later.

Indeed. I just hate the versions hell (as well as dealing with old versions of a language although I happen to love old hardware) so much that I've been ignoring the whole Python until the 3.6 release waiting for the time when one will be able to use all the Python stuff without bothering to learn anything about Python 2.x. It took 10 years of waiting but we are finally here now and now I enjoy Python :-)

Re: Python's New Package Landscape

#156

Post by goerz »

mlthoughts2018 wrote: post #27
Keats wrote: post #18

Earlier quoted context omitted.

I tried a few of the package management in Python recently ( https://www.vincentprouillet.com/blog/overview-package-manag... ) and had the same conclusion with Pipenv. It is way too slow and frankly the UX is not that great either.

I’m disappointed your post did not cover using conda. As the pipenv drama has rolled on, I’ve moved from viewing conda merely as the best user experience in Python environment & package management to instead viewing it as the only serious option for professional scientific computuing work (and quite possibly any professional Python work at all).

Although conda is also getting slower and slower, and now routinely spends 5-20 minutes on dependency resolution even for trivial environments.

Re: Python's New Package Landscape

#157

Post by goerz »

mfe5003 wrote: post #66
Insanity wrote: post #49

Earlier quoted context omitted.

I tend to only use anaconda for "data science work" and not for my small side projects. I "feel" like it is overkill to use anaconda for things unrelated to 'data science' and the likes, but I'm not sure why I feel that way. It kind of makes sense to use for other projects as well since you don't need to import all the things conda offers.

When I have used it, and I have to for a certain project, it is incredibly slow to resolve depenendcies. Enough so that I go for a walk or do something else for 15 minutes while it thinks about whatever it's doing.

Yeah, this is becoming a real problem. This didn't use to happen, but now conda is slow to the point of being unusable if you need to create environments a lot (like during testing)

Re: Python's New Package Landscape

#158

Post by mlthoughts2018 »

goerz wrote: post #156
mlthoughts2018 wrote: post #27

Earlier quoted context omitted.

I’m disappointed your post did not cover using conda. As the pipenv drama has rolled on, I’ve moved from viewing conda merely as the best user experience in Python environment & package management to instead viewing it as the only serious option for professional scientific computuing work (and quite possibly any professional Python work at all).

Although conda is also getting slower and slower, and now routinely spends 5-20 minutes on dependency resolution even for trivial environments.

This is deeply untrue for conda. Even very complex environments build in less than a minute. I can believe there are corner cases where conda is very slow, but claiming conda takes 5 minutes for trivial environments is flat out wrong. Perhaps it is issues with a firewall, VPN connection or something else, absolutely no chance that is from normally executing conda.

Re: Python's New Package Landscape

#159

Post by goerz »

mlthoughts2018 wrote: post #158
goerz wrote: post #156

Earlier quoted context omitted.

Although conda is also getting slower and slower, and now routinely spends 5-20 minutes on dependency resolution even for trivial environments.

This is deeply untrue for conda. Even very complex environments build in less than a minute. I can believe there are corner cases where conda is very slow, but claiming conda takes 5 minutes for trivial environments is flat out wrong. Perhaps it is issues with a firewall, VPN connection or something else, absolutely no chance that is from normally executing conda.

Well, I experience it on a daily basis, and I'm not the only one! It's officially an open issue: https://github.com/conda/conda/issues/7239

Re: Python's New Package Landscape

#160

Post by j88439h84 »

scrollaway wrote: post #10

Published May 11th, 2018. But it's interesting it's popping up again. It's a good explanation of the landscape as of 2018, though Pipenv has since gone in a weird direction. There's a lot of recommendations for it, but I sometimes get the feeling people don't understand what they're recommending, such as replacing some things that work (setup.cfg) by things that don't do the same thing (Pipfile). Man, the Python pack…

Poetry seems promising
Post reply on HN