pip-tools is almost never mentioned because it's boring but great. I always default to it. https://github.com/jazzband/pip-tools
Thanks for mentioning it! How would you compare it to the other tools from the original post?
Overview of Python dependency management tools
111–120 of 184 posts
Re: Overview of Python dependency management tools
#112Earlier quoted context omitted.
> struggling to get Python installed and into their `PATH` ... it's frankly still amateur hour over there But that has been solved on Windows for quite a while hasn't it? Python installs the "py" launcher on the path, which allows you to run whichever version you want of those you have installed. Just type "py" instead of "python". Or "py -3.5-32" to specifically run 32-bit Python 3.5, or "py -0" to list the availabl…
It's gotten a lot better, but we still hit tons of issues with users who don't know what Python version they installed their application in. Oh and of course our "binaries" in Scripts/bin don't seem to show up in the PATH by default. So I get to tell people "py -3.8-64 -m foo" on windows, "foo" everywhere else. This gets much much worse when a new version of Python comes out and we don't support it yet (because of th…
I like Mozilla build system on Windows, you click "start-shell.bat" and it runs console. Python, mercurial, rust - just works, never checked PATH.
https://firefox-source-docs.mozilla.org/setup/windows_build....
Re: Overview of Python dependency management tools
#113pip-tools is almost never mentioned because it's boring but great. I always default to it. https://github.com/jazzband/pip-tools
Yes !! I just create a Makefile target and pip-tools is all I need. I create a requirements.in and that is all. So far never feel that has to be more complicated than that. And when I want to upgrade a package I update the requirements.in if I need to and run `make -B` for this: default: requirements-develop.txt pip install -r requirements-develop.txt requirements.txt: pip-compile -v requirements.in requirements-deve…
Does it pin versions of 2nd degree dependencies too? Like pip freeze would do?
Also, when you remove a package, does it know to clear packages that were its deps and are not needed anymore?
Re: Overview of Python dependency management tools
#114Everytime I read an article about all these tools I really can't help but think what would happened if Linus would have taken over the desktop. All the tools really largely seem to try to poorly replicate Linux package management and the fact that because of this devs now don't care anymore about api stability and not always building against the latest and greatest. I admit a pyenv is nice for testing against differe…
I think that works when you use Python cli tools, but not when you're working on 5 different projects, each running different python version.
Outside of Python 2/3 differences, are Python interpreters not backwards compatible?
In other words, while obviously a program written for 3.3 won't work in 2.7, but will a program written for 3.3 fail to run in 3.8?
If it runs fine, why the need for multiple interpreters? I'd think you'd get by just fine by having the latest 2.x and 3.x installed.
Re: Overview of Python dependency management tools
#115People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…
I think it depends on the use case. If I'm developing my own stuff my peen package management is fine. If trying to run various existing python programs to analyze biology data, I soon run into various problems. Is this a Conda?/ or can I use my Python environment? which version of python? will let me run the thing and what libraries do I need? This breaks in that version? Sometimes I feel that one kinda ok way of do…
> Is this a Conda?/ or can I use my Python environment?
Can you elaborate a bit there? I use conda because I like some of their features over standard virtualenv (being able to specify a python version when i create my venv) - but I've never had a problem running code in env's created by one vs. the other.
Re: Overview of Python dependency management tools
#116Earlier quoted context omitted.
Let me start by saying: I love python, and I love developing in it. It's the "a pleasure to have in class" of languages: phenomenal library support, not too painful to develop in, nice and lightweight so it's easy to throw together test scripts in the shell (contrast that with Java!), easy to specify simple dependencies + install them. (contrast that with C!). That said... if you work on software that is distributed…
Looks like any other package manager: * developers install with language packager * in between install with OS package manager * users install bundle Those who have troubles with pip, gems, cabal, etc should check over options first. Wait, bundlers Gemfile.lock lists installed versions at least ten years, what is "too unbounded" in pip?
It depends! Sometimes I have to lock a dependency at minor releases because every.single.release from the author breaks something new, and I've already worked around the locked version's failings. Sometimes I have to lock a dependency at a major version and everything is fine after that. Usually when the latter happens, eventually the developer releases something that fits within the version bounds and breaks. Sometimes they fix it in the next release, but then I have to deal with a week of bug reports from users that "I couldn't pip install the latest release!". A big complaint I'll with flask/werkzeug app is that something or other broke because they installed something else with strict version requirements alongside it (because the authors of that program have experienced the same bullshit, I assume).
Maybe I'm spoiled from working with cargo and npm (I have almost no ruby experience so I can't comment there), but both of them have way fewer such version conflicts in my experience. Obviously there are tradeoffs and I don't want the node_modules experience for my users, but often it seems that would be a much better experience than pip for everyone. With either of those, I just "npm install" or "cargo install" and all my dependencies end up there working.
You can generate a requirements.txt file using "pip freeze" on a functioning system, but then you have to figure out a way to point users at it instead of using "pip install myapp". Also you might have to do it for each OS since windows vs mac vs linux can have different package dependencies specified, and even if you don't do that, a dependency doing it means you have to account for it.
You can copy+paste the "pip freeze" output into your setup.py and add quotes+commas, but then you're back to breaking side-by-side packages.
So what am I, a developer trying to distribute my command-line application to less-technical users, supposed to do? Distribute two entirely different packages, "myapp-locked" and "myapp"? Tell people to install from a copy+pasted "requirements.txt" file? I've started distributing docker containers that have the application installed via the requirements.txt method, which is fucking stupid but at least the users of that complain less about versioning issues... until the day someone yanks a package I guess.
Re: Overview of Python dependency management tools
#117The trouble is these tools all do different things and aren't really comparable. I wouldn't even include Docker in this kind of thing as it doesn't really do anything on its own. For me, there are two main choices today: * An ensemble of single-purpose tools: pip, venv, pip-tools, setuptools, twine, tox, * An all-in-one tool, for example Poetry, Pipenv or Anaconda (or Miniconda). I prefer the former approach, but if…
I agree with you that Docker should not be there, but the reality is that people us it to replace some other tools (like venv). I wonder why you prefer the former approach.
On the contrary, thanks for having included Docker in that list. It's the obvious answer to so many problems (developing, running and deploying apps, replicating deterministic Python environments, not installing linux dependencies required by Python packages directly on your machine, and so on).
BTW, to comment one of the point you made in the article, it's not that hard to run CUDA inside a container. It's less straightforward but quite well documented. You basically need nvidia-docker [1] on the host and start your containers with the runtime 'nvidia'. docker-compose still doesn't support it officially but there are workarounds. [2] I'm running it on ~50 instances in production and automated all the setup with ansible successfully.
Re: Overview of Python dependency management tools
#118You know how pathlib does for paths and files? Python needs something like that for distribution/versions/import hacking, eh?
Glyph (of Twisted fame, whence pathlib IIRC) pointed this out ages ago: model your domain [in Python] with objects.
Re: Overview of Python dependency management tools
#119> Pipenv or poetry?
If you used pipenv for a complex project with huge dependency tree, or used it for a long time, you definitely run into a blocker issue with it. That is the worst package manager of all, and probably the reason why Python has such a bad reputation in this area. It's because it's fundamentals are terrible.
Just go with Poetry. It's very stable, easy to use, has a superior dependency resolver and way faster than Pipenv.
Re: Overview of Python dependency management tools
#120Dependency management can be pretty overwhelming for a lot of people entering Python. This is especially true in the data science realm, where many don't have a SWE background. Even after you have selected a tool, it can be easy to use it in a poor way. I have recently written a short article on how I use conda in a disciplined way to manage dependencies safely: https://haveagreatdata.com/posts/data-science-python-de…
Python dependency management of packages using C or C++ behind the scenes is really problematic and sometimes, the installation may fail. In this case, a solution is to use Conda or mini conda which provide many pre-compiled packages and also Clang C++ compiler. An alternative way to allow people without software engineering background to play with Python data science and machine learning tool may be providing pre bu…
Docker is definitely an interesting tool for that, but my biggest problems is that I have to teach them Docker, which is a totally new layer of abstraction they haven't seen before.
How do you approach this? How technical are people you prepare Docker images for?