Overview of Python dependency management tools • Hacker News Board

Re: Overview of Python dependency management tools

#41

Post by epistasis » Mon, Jun 01, 2020, 5:27 PM UTC

raziel2p wrote: post #36

throwaway894345 wrote: post #30

Earlier quoted context omitted.

I get that resolving dependencies is a SAT problem and inherently intensive; however, I don't understand why it's so much slower in Python. Is it just that all of these resolvers are implemented in Python (and Python is really that much slower than other languages?), or does Python require you to download an entire package just to determine its dependencies? In the latter case, that seems pretty dumb, right? Like as…

> does Python require you to download an entire package just to determine its dependencies? yes - the standard way of defining dependencies in Python is in setup.py, which has to be invoked as a Python script in order to work. this script may also need to read files from the rest of the project, so you do indeed need to download the whole package to determine its dependencies. even if the Python community were to agr…

It seems that this information should cacheable after an invocation of setup.py, at least for an installation without any extras. And even with extras requested, perhaps.

Or is there any even greater hidden challenge from using setup.py?

Re: Overview of Python dependency management tools

#42

Post by pjs_ » Mon, Jun 01, 2020, 5:35 PM UTC

Anyone installs Conda on my shit I hit the roof... I sympathize with the individual but cannot tolerate the act.

Re: Overview of Python dependency management tools

#43

Post by doteka » Mon, Jun 01, 2020, 5:37 PM UTC

globular-toast wrote: post #11

mariokostelac wrote: post #2

It's 2020, but the python community still has not converged to a small set of sane solutions. It seems to me that Ruby, PHP, JS, and Rust communities have solved the problem.

In the node world I see half of projects telling you how to install it with npm and half with yarn. In Python at least pip is a standard that always works to install, even if it doesn't solve the other problems.

Those are literally equivalent though, anything you can install using yarn you can install using npm. They connect to the same registry and use the same package.json format. Also, pip is relatively recent. Before that, it was a mix of setuptools, easy_install and various manual procedures.

Re: Overview of Python dependency management tools

#44

Post by cycomanic » Mon, Jun 01, 2020, 5:44 PM UTC

Everytime I read an article about all these tools I really can't help but think what would happened if Linus would have taken over the desktop. All the tools really largely seem to try to poorly replicate Linux package management and the fact that because of this devs now don't care anymore about api stability and not always building against the latest and greatest.

I admit a pyenv is nice for testing against different python versions if necessary. But on my Linux systems generally fine with just installing system packages and doing pip install --user for the odd package that is not in the repositories

Re: Overview of Python dependency management tools

#45

Post by mixmastamyk » Mon, Jun 01, 2020, 5:47 PM UTC

smitty1e wrote: post #29

ewidar wrote: post #18

Earlier quoted context omitted.

Hasn't it? pypi is really the de-facto package index, Pipenv/Poetry/Conda are all venv handlers (using the standard venv tools) + dependency graph, and using pip which is standard as well. I would call this a small set of solutions (3), and they are all sane (any will do, just pick one).

Inviting the question: why don't the core devs have a serious bakeoff and bring that functionality into the core distro?

pip, PyPI access, venv, and wheel have all been included for a while.

Re: Overview of Python dependency management tools

#46

Post by slavoingilizov » Mon, Jun 01, 2020, 5:48 PM UTC

The missing ingredient to really, REALLY solve these problems once and for all is an authoritative decision to switch package formats and run the whole dependency resolution stack by the core python language contributor team.

I get backwards compatibility and open-source governance and bla-bla, but the reality is that this cannot be done by a third-party library author and needs to become part of the core stack, including proper support rather than just shipping a tool which covers 90% of cases. It's crazy that apart from venv and pip, nothing else comes with python and you're left on your own.

npm + the registry is part of node

apt-get + registry is part of a normal linux distro

budnler comes with ruby

This is a solved problem elsewhere. What we lack is a fully-supported, agreed upon, working DEFAULT choice, so people don't have to make their own choices. I don't know if not having that DEFAULT is a function of how the python community thinks or its diversity, but it's painful to watch. I've almost given up myself and seen many newcomers give up because of a trivial problem like this.

Re: Overview of Python dependency management tools

#47

Post by edsac_xyzw » Mon, Jun 01, 2020, 5:51 PM UTC

akbo wrote: post #34

Dependency management can be pretty overwhelming for a lot of people entering Python. This is especially true in the data science realm, where many don't have a SWE background. Even after you have selected a tool, it can be easy to use it in a poor way. I have recently written a short article on how I use conda in a disciplined way to manage dependencies safely: https://haveagreatdata.com/posts/data-science-python-de…

Python dependency management of packages using C or C++ behind the scenes is really problematic and sometimes, the installation may fail. In this case, a solution is to use Conda or mini conda which provide many pre-compiled packages and also Clang C++ compiler.

An alternative way to allow people without software engineering background to play with Python data science and machine learning tool may be providing pre built Docker images with everything pre-installed which may save one from configuration trouble.

Docker is also useful for learning about new programming languages without installing anything. With just one command $ docker "run --rm -it julia-image", one can get a Docker image containing a GOLang compiler; a Julia language installation; a Rust development environment and everything else. Docker is really a wonderful tool.

Re: Overview of Python dependency management tools

#48

Post by franciscop » Mon, Jun 01, 2020, 5:52 PM UTC

For some of the problems that Node.js and JS at large have with a centralized package manager, I for one am very happy that it's not in the python situation. 100% of the packages I've tried to install in the last 3+ years are simply `npm install PKG`.

Re: Overview of Python dependency management tools

#49

Post by xapata » Mon, Jun 01, 2020, 5:54 PM UTC

franciscop wrote: post #48

For some of the problems that Node.js and JS at large have with a centralized package manager, I for one am very happy that it's not in the python situation. 100% of the packages I've tried to install in the last 3+ years are simply `npm install PKG`.

Do popular Node packages rely on C and Fortran?

Re: Overview of Python dependency management tools

#50

Post by madelyn » Mon, Jun 01, 2020, 5:58 PM UTC

People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me?

I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and it's not been too frustrating.

A lot of the issues people have with Python packaging seem like they can get replaced with a couple shell aliases. Dependency hell with too many dependencies becomes unruly in any package manager I've tried.

Is the "silent majority" just productive with the status quo and getting work done with Python behind the scenes? Why is my experience apparently so atypical?