Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

141–150 of 174 posts

Re: Python's New Package Landscape

#141

Post by scrollaway »

Drdrdrq wrote: post #138
Bogdanp wrote: post #19

Earlier quoted context omitted.

It takes about 2 minutes (feels like 5!) on my 2016 MBP to install 102 dependencies. Doing that in Docker takes about 1.5x the time. I haven't seen it take 20-30 minutes, but 2-3 minutes is still obscenely slow in my view.

There is no reason for Docker to be slower, it must be some kind of configuration issue. Containers are basically just processes and there is virtually no difference in execution times.

As I said in my other reply, Docker is more likely to have ephemeral storage for the cache. So every single lock it'll re-download the package. Whereas locally, you're likely to still have the packages cached.

This can make a difference of tens of minutes for some packages which have a 1 gigabyte (!!!) download.

Re: Python's New Package Landscape

#142

Post by reilly3000 »

Why is it that Python is geared towards archiving packages at the site level by default while npm, composer, et all tend towards including packages in the project's folder? Is it convention from a time when disk space was less plentiful?

Re: Python's New Package Landscape

#144

Post by gary_bernhardt »

reilly3000 wrote: post #142

Why is it that Python is geared towards archiving packages at the site level by default while npm, composer, et all tend towards including packages in the project's folder? Is it convention from a time when disk space was less plentiful?

"Why" is hard, but Python's packaging system was created when global installation was just how packaging worked. There may have been exceptions, but the first local package installation tool I knew of was workingenv (2006). It was the predecessor of virtualenv, which I think led directly to pipenv.

Re: Python's New Package Landscape

#146

Post by binalpatel »

I find conda far and above the best tool to manage python packages and dependencies. Being able to concisely contain all Python and binary dependencies together is invaluable.

I recently wrote about it as blog post, using conda within containers has solved almost every pain point we had with python packaging and how to get things into production reliably.

Re: Python's New Package Landscape

#147

Post by orf »

cik wrote: post #94

We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it take…

Slow dependency resolution is a known problem and will be fixed in a newer release. The reason is not entirely pipenvs fault (curse be to setup.py), and the pipenv maintainers are mimicking the somewhat novel method twine uses.

Re: Python's New Package Landscape

#148

Post by orf »

binalpatel wrote: post #146

I find conda far and above the best tool to manage python packages and dependencies. Being able to concisely contain all Python and binary dependencies together is invaluable. I recently wrote about it as blog post, using conda within containers has solved almost every pain point we had with python packaging and how to get things into production reliably.

Counterpoint: it's terrible. A lack of a lockfile is a killer, plus it not really fitting well with the general ecosystem (it's not really a python dependency manager). It's an all or nothing tool, which sucks to be honest

Now most projects have wheels pip is pretty damn good.

The conda CLI is also just terrible. It's good for ad-hoc research, but for big deployments? No thanks, I've had enough pain using it.

Re: Python's New Package Landscape

#149

Post by erikb »

If you only think in Python then packaging really might be a painpoint. But honestly if we look at other languages it's not so bad actually, very specifically calling out Golang here because it claimed exactly this topic as an initial design goal and to this day has basically failed at delivering it.

There are even languages like C++ where the community as a (w)hole has given up on that topic and instead opts for completely building every tool by building the underlying libraries up first manually.

Considering all this, who can actually beat Python at this point? Java maybe? Is Ruby still competing? How is NodeJS doing?

Currently with what I see around me (mostly Go and C++) I don't feel too bad about setuptools+pip+virtualenv anymore.

Re: Python's New Package Landscape

#150

Post by Serow225 »

jonafato wrote: post #130
Serow225 wrote: post #119

Earlier quoted context omitted.

Sure, but couldn't it have been given some BFDL/high-level leadership importance in the last five years to rein in the craziness?

The PyPA team has done a lot over the past five years. The changelog for pip ( https://pip.pypa.io/en/stable/news/ ) contains quite a bit, PyPI was migrated to Warehouse, and there have been several PEPs focused on improving the packaging situation. A lot of these ideas come from various people in the community and get formalized as official recommendations or tools, but these things take time, especially accounting…

Understood, I guess I'm wondering why it hasn't been possible to cull more of the less-successful attempts, or at least make it obvious to newer users what is legacy. As an outsider/newer person to Python, the number of package mgmt options to consider is vast and confusing, it would be helpful if there was one (or a few) more "blessed" solutions :)
Post reply on HN