Overview of Python dependency management tools • Hacker News Board

Re: Overview of Python dependency management tools

#81

Post by ssully » Mon, Jun 01, 2020, 6:47 PM UTC

IshKebab wrote: post #68

madelyn wrote: post #50

People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…

I don't think your experience is atypical but I do think your acceptance of something that is quite awful is fairly atypical. It's also possible that you use Python on Linux, where it is at least tolerable. Try again on Windows.

I develop Python exclusively on Windows (that then is deployed on Linux) and my experience is identical to the original poster. It's not a perfect system, but it's good enough and I have dealt with dependency management systems.

Re: Overview of Python dependency management tools

#82

Post by polyrand » Mon, Jun 01, 2020, 6:48 PM UTC

mumblemumble wrote: post #62

vmsp wrote: post #23

pip-tools is almost never mentioned because it's boring but great. I always default to it. https://github.com/jazzband/pip-tools

I came to the comments to say exactly this. There's a decent summary of why someone might still prefer pip-tools even in a world where pipenv and poetry exist here: https://hynek.me/articles/python-app-deps-2018/ For my purposes, the primary downside of this approach is that adding dependencies takes slightly more effort, because you have to edit a file and then execute a shell command, rather than just executing a s…

I fully agree. But I see editing the file manually an advantage. I can pip install whatever I want and then I only need to worry about having a clean requirements.in file.

With that, I know the compiled requirements.txt will only have what I need. Now it is just pip install -r requirements.txt or pip-sync.

Re: Overview of Python dependency management tools

#83

Post by [deleted] » Mon, Jun 01, 2020, 6:51 PM UTC

luhn wrote: post #75

bvar10 wrote: post #64

Serious question: What is the difference between virtual environments and just having several Python installs like: /home/foo/a/usr/bin/python3 /home/foo/b/usr/bin/python2 Python is so fast to compile and install that I just install as many throwaway Pythons as needed. I do not recall any isolation issues between those installs, unlike with conda or venv, which are both subtly broken on occasion. But I dislike opaque…

That basically is what a venv is, an entirely separate Python install. Some files are linked rather than being copied, but it looks the same. venv gets you a couple extra conveniences, like the activation script. I wouldn't call venv "opaque automation," there's not much magic going on there.

[deleted]

Re: Overview of Python dependency management tools

#84

Post by nwatson » Mon, Jun 01, 2020, 6:55 PM UTC

boarnoah wrote: post #73

Could someone summarize the issues with Pipenv (and by Extension Poetry). Been using them happily for the last few years, didn't know people disliked them. With Pipenv, last year ownership switched from the Request's lib owner to the Pypa, so more or less an officially blessed solution. The only downside on this thread that I could understand so far is that it might be slow to install dependencies on larger projects,…

I've been using pipenv happily for a few years now, but on projects that don't have a huge number of dependencies (Django, DRF, MySQL/Postgres, AWS, Kubernetes, a few other random libraries), and haven't seen too much slowness. I suppose data-science projects with large dependencies that pull in many other dependencies might have more issues.

I've been the person to document setting up development environments for others in macOS (and Homebrew) with a view to deploying in Linux, and pipenv (and pyenv, and Docker/docker-compose for setting up software context/datasets) definitely overall minimized the complexity for those configuring their dev environments.

(EDIT: documenting dev enviroments)

Re: Overview of Python dependency management tools

#85

Post by edsac_xyzw » Mon, Jun 01, 2020, 6:56 PM UTC

globular-toast wrote: post #3

The trouble is these tools all do different things and aren't really comparable. I wouldn't even include Docker in this kind of thing as it doesn't really do anything on its own. For me, there are two main choices today: * An ensemble of single-purpose tools: pip, venv, pip-tools, setuptools, twine, tox, * An all-in-one tool, for example Poetry, Pipenv or Anaconda (or Miniconda). I prefer the former approach, but if…

Besides the dependency management, another major problem of Python is the deployment. Although Docker is not a dependency management tool, it can be used as a deployment tool which encapsulates the application to be deployed, python runtime and all other shared libraries dependencies alongside the configuration. Another deployment tool that is worth mentioning is Pyinstaller. It can pack the python runtime, the application and all dependencies into a single native executable. Pyinstaller is better for Desktop applications and for building single file executable applications.

Re: Overview of Python dependency management tools

#86

Post by d4mi3n » Mon, Jun 01, 2020, 6:58 PM UTC

drcongo wrote: post #71

j88439h84 wrote: post #59

Earlier quoted context omitted.

Poetry is better than Pipenv by a mile. It solves almost all of the problems, and the remaining ones are already on the Poetry roadmap.

Oh, totally agree. But it also seems to have a lot more problems than it did two years ago.

I'd chalk this up to dependency management and resolution being a hard problem.

Ruby's bundler had these exact same issues 5 or so years ago. I remember attending a talk by on Bundler run by it's core devs and asking about how they make dep resolution faster. Turns out that it was never really a solved problem there either, Bundler just uses a bunch of heuristics to avoid cases like the 18 minute `Pipefile.lock` described above.

Re: Overview of Python dependency management tools

#87

Post by steveklabnik » Mon, Jun 01, 2020, 6:59 PM UTC

slavoingilizov wrote: post #46

The missing ingredient to really, REALLY solve these problems once and for all is an authoritative decision to switch package formats and run the whole dependency resolution stack by the core python language contributor team. I get backwards compatibility and open-source governance and bla-bla, but the reality is that this cannot be done by a third-party library author and needs to become part of the core stack, incl…

While npm comes with node, and bundler comes with Ruby, the governance of these projects/tools are separate from the language.

Re: Overview of Python dependency management tools

#88

Post by franciscop » Mon, Jun 01, 2020, 7:01 PM UTC

d0mine wrote: post #79

franciscop wrote: post #60

Earlier quoted context omitted.

Do you mean pip3 install ? Also tried that, didn't work. Had to learn about Python versions, pip vs pip3 versions, pipenv, conda, how an old python package doesn't work with a modern Python package, etc. All I was trying is to combine tensorflow lite with opencv IIRC. Just look at the installation instructions: - https://www.tensorflow.org/install/pip - https://www.tensorflow.org/lite/guide/python - https://docs.open…

The instructions that you've link use `pip install tensorflow`

Yes, but those instructions did not work for me. I am not sure if they didn't work because the required previous libraries were wrong somehow (which would just be `dependencies` in package.json), a different version (non-issue with `dependencies`), my python version was wrong (non-issue with `engines`), etc.

Re: Overview of Python dependency management tools

#89

Post by sosodev » Mon, Jun 01, 2020, 7:07 PM UTC

xapata wrote: post #40

CharlieBlack11 wrote: post #25

Tbh, this is one of the reasons why I moved away from Python to Ruby for my side projects.

Are bundler, rvm, and rbenv not as confusing?

Definitely not. In Ruby I can immediately understand how to run a correct copy of any project because they all use bundler. Furthermore rbenv makes switching between specific Ruby versions for those apps trivial.

Re: Overview of Python dependency management tools

#90

Post by mumblemumble » Mon, Jun 01, 2020, 7:11 PM UTC

polyrand wrote: post #82

mumblemumble wrote: post #62

Earlier quoted context omitted.

I came to the comments to say exactly this. There's a decent summary of why someone might still prefer pip-tools even in a world where pipenv and poetry exist here: https://hynek.me/articles/python-app-deps-2018/ For my purposes, the primary downside of this approach is that adding dependencies takes slightly more effort, because you have to edit a file and then execute a shell command, rather than just executing a s…

I fully agree. But I see editing the file manually an advantage. I can pip install whatever I want and then I only need to worry about having a clean requirements.in file. With that, I know the compiled requirements.txt will only have what I need. Now it is just pip install -r requirements.txt or pip-sync.

That's a very good point. I don't think it had even occurred to me that it would be more difficult with pipenv or poetry. But, admittedly, I haven't made it far into either of them - a little noodling around, just enough to figure out that they don't really address any pain points for me.