[deleted]
Overview of Python dependency management tools
101–110 of 184 posts
Re: Overview of Python dependency management tools
#102The trouble is these tools all do different things and aren't really comparable. I wouldn't even include Docker in this kind of thing as it doesn't really do anything on its own. For me, there are two main choices today: * An ensemble of single-purpose tools: pip, venv, pip-tools, setuptools, twine, tox, * An all-in-one tool, for example Poetry, Pipenv or Anaconda (or Miniconda). I prefer the former approach, but if…
Besides the dependency management, another major problem of Python is the deployment. Although Docker is not a dependency management tool, it can be used as a deployment tool which encapsulates the application to be deployed, python runtime and all other shared libraries dependencies alongside the configuration. Another deployment tool that is worth mentioning is Pyinstaller. It can pack the python runtime, the appli…
Docker is an invaluable tool, but it's a general purpose one. It's good for Python because it's good for everything.
I have used Pyinstaller. It's good but it's a bit too magical for my tastes.
Re: Overview of Python dependency management tools
#103For some of the problems that Node.js and JS at large have with a centralized package manager, I for one am very happy that it's not in the python situation. 100% of the packages I've tried to install in the last 3+ years are simply `npm install PKG`.
Do popular Node packages rely on C and Fortran?
I don’t recall any popular Node package relying on Fortran, but there are two popular packages that rely on C: fsevents and node-sass
It works on macOS and Linux without any issues. Windows usually requires some extra steps to setup node-gyp
Re: Overview of Python dependency management tools
#104Earlier quoted context omitted.
Let me start by saying: I love python, and I love developing in it. It's the "a pleasure to have in class" of languages: phenomenal library support, not too painful to develop in, nice and lightweight so it's easy to throw together test scripts in the shell (contrast that with Java!), easy to specify simple dependencies + install them. (contrast that with C!). That said... if you work on software that is distributed…
> struggling to get Python installed and into their `PATH` ... it's frankly still amateur hour over there But that has been solved on Windows for quite a while hasn't it? Python installs the "py" launcher on the path, which allows you to run whichever version you want of those you have installed. Just type "py" instead of "python". Or "py -3.5-32" to specifically run 32-bit Python 3.5, or "py -0" to list the availabl…
It's gotten a lot better, but we still hit tons of issues with users who don't know what Python version they installed their application in. Oh and of course our "binaries" in Scripts/bin don't seem to show up in the PATH by default. So I get to tell people "py -3.8-64 -m foo" on windows, "foo" everywhere else.
This gets much much worse when a new version of Python comes out and we don't support it yet (because of the build system issues I mentioned). I spent several weeks teaching people how to uninstall 3.8 and install 3.7 before we finally got a functioning package out for 3.8.
Re: Overview of Python dependency management tools
#105Earlier quoted context omitted.
Plain old pip and venv can do that. just "pip freeze >requirements.txt" and elsewhere "pip install -r requirements.txt", inside venvs.
I think that will end up installing the subdependency version of whatever is last in the requirements.txt. You need a dependency resolver to deal with problems with conflicting versions. More details here: https://medium.com/knerd/the-nine-circles-of-python-dependen...
pip handles the simple cases: if you install a new pkgA that depends on 'pkgB=3'. You probably want for pip (or similar) to figure out that an older version of 'pkgC' is compatible with 'pkgB>2'.
But I actually don't want it to be too smart. Better to keep your dependencies minimal and explicit, and manually specify older 'pkgC' if you need to. I have a few non-trivial services in production, the most complex one with 16 total dependencies + sub-dependencies. That is quite manageable.
So, I strongly recommend manually curating the most appropriate versions of the few tastefully chosen dependencies you really need. Then, pip+venv can easily reproduce that exact set of dependencies anytime. I also do something very similar to this with C applications, and Go. Sub-dependencies should be a big factor in how you choose your direct dependencies.
Re: Overview of Python dependency management tools
#106Earlier quoted context omitted.
To be blunt, maybe you just don't know what you're missing out on? Of course, Python's package management system works and is merely an annoyance to those of us who are used to more modern package managers. By the way, your comment reminded me a of this classic: https://news.ycombinator.com/item?id=9224 :)
I mean, possibly? What's considered the gold standard in package management these days? I use yarn for managing javascript dependencies and do a lot of work with Cargo too. The community seems to love both these tools outside of slow compile and install times.
Cargo is my ideal, but really anything that doesn't make me manage virtualenvs or take 30 minutes to resolve dependencies. Note that "managing my own virtualenvs" is tricky because you have to make sure everyone has all of the same versions of the same dependencies in their virtualenv across your entire team (including production). I'm sure there are workflows that allow for this (probably with some tradeoffs), but we haven't figured it out. For a while we used Docker, but performance degraded exponentially as our test base grew (Docker for Mac filesystem problems, probably). Eventually we settled on pantsbuild.org which has a lot of problems, is super buggy, no one can figure out its plugin architecture, etc but as long as you stay on the happy path it generally works okay which puts it in one of the ballparks between any other Python dependency management scheme I've tried and Go/Rust/etc package management.
Re: Overview of Python dependency management tools
#107People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…
Let me start by saying: I love python, and I love developing in it. It's the "a pleasure to have in class" of languages: phenomenal library support, not too painful to develop in, nice and lightweight so it's easy to throw together test scripts in the shell (contrast that with Java!), easy to specify simple dependencies + install them. (contrast that with C!). That said... if you work on software that is distributed…
Looks like any other package manager:
* developers install with language packager
* in between install with OS package manager
* users install bundle
Those who have troubles with pip, gems, cabal, etc should check over options first.
Wait, bundlers Gemfile.lock lists installed versions at least ten years, what is "too unbounded" in pip?
Re: Overview of Python dependency management tools
#108People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…
When it comes to shipping Python server-side apps, Pipenv is a godsend. Before discovering it, I had 3 requirements.txt files (common, dev, prod) which I had to edit manually. This often meant forgetting to include something that I just installed and only finding out after a full round of QA.
It also meant a separate couple of steps for full-tree dependency freezing which never worked quite properly anyways. Pipenv just....works. Dependencies are saved as I install them, I only have to deal with the top-level ones, but the whole tree is locked.
Re: Overview of Python dependency management tools
#109Earlier quoted context omitted.
I'm only familiar with Python, Javascript and Rust. It seems to me that Rust is the only one that has "solved" this problem. I dont think there are any real Python devs who thinks dependency management is solved. However, why would you claim Javascript has a good solution? The inconsistencies between node and web dev is odd at best. Babel compilation is annoying and slow. Are we even standardized on webpack yet? Can…
> As pioneers in dependency management Maybe a noob question, but how come they are pioneers? Weren't there languages with package systems before Python and JS?
My first encounter with something like a package manager was CPAN for Perl... maybe they were the pioneers
Re: Overview of Python dependency management tools
#110Oof, that footnote: > It’s 2020, but reliably compiling software from source, on different computer setups, is still an unsolved problem. There are no good ways to manage different versions of compilers, different versions of libraries needed to compile the main program etc. I wonder how much stuff like this has to do with python's popularity. When I have opaque issues like "libaslkdjfasf.so is angry with you and/or…
I think this is not a problem specific to python packages, but a general problem of how we compile C/C++ software. There is no concept of packages and compiling one thing often requires installing a -dev package of some other library.
The issue is that lack of packaging C/C++ world spreads to all other communities that depend on them.