Live data from Hacker News

Overview of Python dependency management tools

modelpredict.com

31–40 of 184 posts

Re: Overview of Python dependency management tools

#31
I like this overview. However, it points to a fundamental problem with Python environment, going much against its own credo:

"There should be one - and preferably only one - obvious way to do it." - The Zen of Python; see also https://xkcd.com/1987/.

When it comes to the package, environment and dependency management, I think that ironically JavaScript environment is light years head, vide: https://p.migdal.pl/2020/03/02/types-tests-typescript.html

Re: Overview of Python dependency management tools

#32
post #17

Earlier quoted context omitted.

> Can anyone say with a straight face that getting a new JS dev caught up on what all these different parts to compile a JS program is a solved problem? You are intermixing dependency management with build tools. Webpack and Babel have very little to do with dependency management.

I loosely included Babel and Webpack as part of dependency management since Python does not have a similar compile step. I don't think it is unfair since they fall into the same category of wtf when trying to get your dev environment working. With that said, I should have included Yarn and Npm with their own problems. I can't remember how often I solved my dependency problems with rm -rf node_modules.

Python does have a compile step if your packages contain C extensions and aren't bundled into binary wheels. This shit tends to only work because of a whole lot of effort by package maintainers to make sure it compiles on a handful of supported systems, but if you aren't running such a system you get to debug a bunch of C dependency errors buried under a mile of gcc warnings and other make output.

Re: Overview of Python dependency management tools

#33

This is a good overview of something that took me an annoyingly long time to learn. My personal preference is to keep things simple with pyenv, venv, and pip. Tangentially related is the tool tox [1], which is often used to run a test suite inside of virtual environments created by venv, on multiple versions of Python managed by pyenv. Now if only setuptools could work well without hackery... [1]: https://tox.readthe…

Wouldn’t you still need something like pip-tools to lock down subdependencies and handle conflicts?

Plain old pip and venv can do that. just "pip freeze >requirements.txt" and elsewhere "pip install -r requirements.txt", inside venvs.

Re: Overview of Python dependency management tools

#34
Dependency management can be pretty overwhelming for a lot of people entering Python. This is especially true in the data science realm, where many don't have a SWE background. Even after you have selected a tool, it can be easy to use it in a poor way. I have recently written a short article on how I use conda in a disciplined way to manage dependencies safely: https://haveagreatdata.com/posts/data-science-python-depende...

Re: Overview of Python dependency management tools

#35
post #33

Earlier quoted context omitted.

Wouldn’t you still need something like pip-tools to lock down subdependencies and handle conflicts?

Plain old pip and venv can do that. just "pip freeze >requirements.txt" and elsewhere "pip install -r requirements.txt", inside venvs.

I think that will end up installing the subdependency version of whatever is last in the requirements.txt. You need a dependency resolver to deal with problems with conflicting versions.

More details here: https://medium.com/knerd/the-nine-circles-of-python-dependen...

Re: Overview of Python dependency management tools

#36
post #15

Every attempt to solve this problem in Python seems to eventually end up in a pretty terrible place. Pipenv got off to a great start but got slower and slower to the point that it was more painful to use than not. Poetry (which is still my preferred option) started off with something seemingly beautifully thought through, and very fast too. But after only a few version updates, it seems to be hitting the same problem…

I get that resolving dependencies is a SAT problem and inherently intensive; however, I don't understand why it's so much slower in Python. Is it just that all of these resolvers are implemented in Python (and Python is really that much slower than other languages?), or does Python require you to download an entire package just to determine its dependencies? In the latter case, that seems pretty dumb, right? Like as…

> does Python require you to download an entire package just to determine its dependencies?

yes - the standard way of defining dependencies in Python is in setup.py, which has to be invoked as a Python script in order to work. this script may also need to read files from the rest of the project, so you do indeed need to download the whole package to determine its dependencies.

even if the Python community were to agree on a new configuration format tomorrow, there would still be a ton of packages out there that wouldn't migrate for years.

Re: Overview of Python dependency management tools

#37

It's 2020, but the python community still has not converged to a small set of sane solutions. It seems to me that Ruby, PHP, JS, and Rust communities have solved the problem.

.NET has it’s issues mostly due to the historical mess of frameworks, but with development converging around .NET Core things have been getting a lot better.

I think the environment system in python is a confusing design flaw that could have been avoided with project specific installations. I vastly prefer installing packages on a project-by-project basis. Python introduces dependency nightmares because two projects with different needs end up using the same central local package source unless you set up different environments. So when you install an package foo for project B, package bar might stop working for project A due a dependency on an earlier version of foo.

Re: Overview of Python dependency management tools

#39
It's worth noting that on Linux it's slightly different because most of the popular libraries can be installed with the system package manager (no problem of dependency management, updates, ...), I rely on alternative solutions only when I want to use a version of a library different from the one shipped with the package manager (which is not that frequent with fast paced distros like Fedora) or when the library is not packaged.
Post reply on HN