Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overview of Python dependency management tools

modelpredict.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: Overview of Python dependency management tools

#121

Post by mjw1007 »

autograd wrote: post #113
mariocesar wrote: post #63

Earlier quoted context omitted.

Yes !! I just create a Makefile target and pip-tools is all I need. I create a requirements.in and that is all. So far never feel that has to be more complicated than that. And when I want to upgrade a package I update the requirements.in if I need to and run `make -B` for this: default: requirements-develop.txt pip install -r requirements-develop.txt requirements.txt: pip-compile -v requirements.in requirements-deve…

Does it pin versions of 2nd degree dependencies too? Like pip freeze would do? Also, when you remove a package, does it know to clear packages that were its deps and are not needed anymore?

pip-tools can do both of those, yes.

For the second, pip-compile computes the new requirements.txt (which is effectively the lockfile) from scratch, and pip-sync (not shown in that Makefile fragment) removes packages that are no longer listed there.

Re: Overview of Python dependency management tools

#122

Post by mariokostelac »

Sohcahtoa82 wrote: post #114
mariokostelac wrote: post #95

Earlier quoted context omitted.

I think that works when you use Python cli tools, but not when you're working on 5 different projects, each running different python version.

Outside of Python 2/3 differences, are Python interpreters not backwards compatible? In other words, while obviously a program written for 3.3 won't work in 2.7, but will a program written for 3.3 fail to run in 3.8? If it runs fine, why the need for multiple interpreters? I'd think you'd get by just fine by having the latest 2.x and 3.x installed.

It's just about backward compatibility. If you don't run the exact version of python that's running in production, how do you know that you're not using some method that does not exist in production yet (because you run older version there)?

Also, libraries with binary component often have to be compiled against specific version of python.

Re: Overview of Python dependency management tools

#124

Post by Groxx »

mariokostelac wrote: post #111
vmsp wrote: post #23

pip-tools is almost never mentioned because it's boring but great. I always default to it. https://github.com/jazzband/pip-tools

Thanks for mentioning it! How would you compare it to the other tools from the original post?

Pipenv is almost exactly pip-tools + venv + a custom UI over the two. And slightly easier / more automatic venv management.

From the times I've looked at it (almost a year ago and older): pip-tools is / was the core of Pipenv... but it has been a fair distance ahead in terms of bug fixes and usable output for diagnosing conflicts. It seemed like pipenv forked from pip-tools a couple years prior, and didn't keep up.

Given that, I've been a happy user of v(irtual)env and pip-tools. Pipenv has remained on the "maybe some day, when it's better" side of things, since I think it does have potential.

Re: Overview of Python dependency management tools

#125

Post by claytonjy »

raziel2p wrote: post #36
throwaway894345 wrote: post #30

Earlier quoted context omitted.

I get that resolving dependencies is a SAT problem and inherently intensive; however, I don't understand why it's so much slower in Python. Is it just that all of these resolvers are implemented in Python (and Python is really that much slower than other languages?), or does Python require you to download an entire package just to determine its dependencies? In the latter case, that seems pretty dumb, right? Like as…

> does Python require you to download an entire package just to determine its dependencies? yes - the standard way of defining dependencies in Python is in setup.py, which has to be invoked as a Python script in order to work. this script may also need to read files from the rest of the project, so you do indeed need to download the whole package to determine its dependencies. even if the Python community were to agr…

Does this imply that poetry will "solve" dependencies quicker the more of the dependencies use `pyproject.toml`? Or is that "hidden" once something is sent to PyPI anyways?

Re: Overview of Python dependency management tools

#126

Post by a_cool_username »

Sohcahtoa82 wrote: post #114
mariokostelac wrote: post #95

Earlier quoted context omitted.

I think that works when you use Python cli tools, but not when you're working on 5 different projects, each running different python version.

Outside of Python 2/3 differences, are Python interpreters not backwards compatible? In other words, while obviously a program written for 3.3 won't work in 2.7, but will a program written for 3.3 fail to run in 3.8? If it runs fine, why the need for multiple interpreters? I'd think you'd get by just fine by having the latest 2.x and 3.x installed.

Sometimes! A very simple example is code that uses "async" as a variable name. It became a keyword in 3.5, which was an enormous pain in the ass.

Re: Overview of Python dependency management tools

#127

Post by autograd »

mjw1007 wrote: post #121
autograd wrote: post #113

Earlier quoted context omitted.

Does it pin versions of 2nd degree dependencies too? Like pip freeze would do? Also, when you remove a package, does it know to clear packages that were its deps and are not needed anymore?

pip-tools can do both of those, yes. For the second, pip-compile computes the new requirements.txt (which is effectively the lockfile) from scratch, and pip-sync (not shown in that Makefile fragment) removes packages that are no longer listed there.

Thanks a lot for the reply. I'll include it some time soon in the article update

Re: Overview of Python dependency management tools

#128

Post by pas »

madelyn wrote: post #70
olalonde wrote: post #65

Earlier quoted context omitted.

To be blunt, maybe you just don't know what you're missing out on? Of course, Python's package management system works and is merely an annoyance to those of us who are used to more modern package managers. By the way, your comment reminded me a of this classic: https://news.ycombinator.com/item?id=9224 :)

I mean, possibly? What's considered the gold standard in package management these days? I use yarn for managing javascript dependencies and do a lot of work with Cargo too. The community seems to love both these tools outside of slow compile and install times.

yarn. fast and correct. every package.json is a virtualenv, but you still have a global cache and it just symlinks. (sort of like the global wheels cache for pip, but even more deduplication.)

cargo is great because it manages the build flow, it's extensible (clippy) - the global cache thing is a bit harder, because of rust package features (and other knobs like RUSTFLAGS), and it's not done by default, but it's as easy as setting RUST_TARGET_DIR as far as I know.

Re: Overview of Python dependency management tools

#129

Post by pacifika »

edsac_xyzw wrote: post #85
globular-toast wrote: post #3

The trouble is these tools all do different things and aren't really comparable. I wouldn't even include Docker in this kind of thing as it doesn't really do anything on its own. For me, there are two main choices today: * An ensemble of single-purpose tools: pip, venv, pip-tools, setuptools, twine, tox, * An all-in-one tool, for example Poetry, Pipenv or Anaconda (or Miniconda). I prefer the former approach, but if…

Besides the dependency management, another major problem of Python is the deployment. Although Docker is not a dependency management tool, it can be used as a deployment tool which encapsulates the application to be deployed, python runtime and all other shared libraries dependencies alongside the configuration. Another deployment tool that is worth mentioning is Pyinstaller. It can pack the python runtime, the appli…

Unless you’re on macOS where you have to manually edit the pyinstaller package to comment out or point to an updated Ntlk data dump.

Re: Overview of Python dependency management tools

#130

Post by virusduck »

franciscop wrote: post #60
d0mine wrote: post #51

Earlier quoted context omitted.

pip install works even on iPhone (for pure Python packages in Pythonista for iOS)

Do you mean pip3 install ? Also tried that, didn't work. Had to learn about Python versions, pip vs pip3 versions, pipenv, conda, how an old python package doesn't work with a modern Python package, etc. All I was trying is to combine tensorflow lite with opencv IIRC. Just look at the installation instructions: - https://www.tensorflow.org/install/pip - https://www.tensorflow.org/lite/guide/python - https://docs.open…

DONT FORGET sudo pip
Post reply on HN