This is in 2020 and should be marked as such.
Pipenv: Promises a Lot, Delivers Little (2020)
11–20 of 68 posts
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#12It's a bit of an old post, and yes Pipenv is not the go-to tool anymore. pip-tools is okay for people that really, really love their requirements.txt; otherwise we tend to go with Poetry at work. Any folks having a good experience with PDM https://github.com/pdm-project/pdm ?
How is the maturity/stability of Poetry these days? I despise Pipenv, and was hoping to push for a switch to Poetry at my place of work a couple years ago, but I ran into blocking bugs across multiple versions (latest N versions affected by bug A, prior M versions affected by bug B). Had to chock it up to "not yet mature enough" and resign myself to the absurd lock times and countless terrible behaviors of Pipenv.
We use poetry for all python projects. I haven't seen an actual poetry internal bug in quite a while, but using poetry effectively does require one keep some things in mind that are probably non-obvious to newcomers:
1. Poetry's default assumption on packages respecting semver simply does not hold up in reality. There are very few packages actually sticking to semver. Thus the `^x.y.z` default version range is quite often too loose. I've found that using `~x.y.z` for most packages is far more stable.
2. Imho, `poetry update` is a footgun. Without specifier, it will attempt to update the entire dependency tree. Not only is this slow, together with 1) it's all too likely one ends up with dependencies that actually are incompatible at runtime. I'd much rather have a `poetry update --all` flag instead for the rare instance I do want to update everything. The default behaviour should be to require a list of packages to update.
3. There are some common packages that cause very long resolution times if they are not restricted. Case in point: boto3. Even if one doesn't use boto3 oneself, it's very likely a transient dependency. Many packages simply specify `'*'` as their version dependency (they shouldn't, but it's the unfortunate reality many do). This will cause poetry to consider every possible boto3 version. With hundreds of versions - boto3 has a release every other day - this gets unwieldy fast. So I often end up specifying boto3 myself with some sensible range in my toml file, even when it's not a strict dependency of my own project.
4. The datascience ecosystem needs particular attention. Best to simply pin those, as every pandas update is guaranteed to break something. ABI changes to numpy are a particular nightmare. This is again due to too many packages simply specifying `'*'` for their numpy dependency. Which is further complicated by the fact that most don't distinguish between build-time dependencies and run-time dependencies. The numpy ABI is only forward compatible. Hence one should build with the oldest supported numpy[0].
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#13I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but…
The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.
Upgrading dependencies is also a way for me to resolve issues, so I don't fight it.
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#14I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but…
The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.
pip freeze > stable-reqs.txt ??
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#15I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but…
You don't need virtual environments when using npm or composer or cargo.
They also create lock files with hashes by default.
pip still doesn't use lock files. It's subpar compared to other ecosystems.
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#16Earlier quoted context omitted.
The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.
pip freeze > stable-reqs.txt ??
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#17I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but…
You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.
I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly.
If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program.
Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#18Earlier quoted context omitted.
You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.
I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly. If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program. Using lock files for libraries should absolutely never happen, you should at most fix your dependen…
I agree, but they also include hashes.
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#19I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but…
The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.
That's a feature, you should be using the latest minor/patch release when developing. If you really need to lock a dependency to a specific version it is possible, but you shouldn't be doing it. If a dependency is constantly breaking on minor/patch releases you shouldn't be using it.
I think it's a lot of javascript developers who end up having this problem, and all I can say is that the python package ecosystem is not the same. Please don't lock your dependencies to specific minor/patch versions, and please don't use so many dependencies that it becomes tedious to deal with changes. Especially when you're writing a library. If you're locking to anything other than a major release or a minimum minor/patch revision when producing a library things have gone very wrong.
Re: Pipenv: Promises a Lot, Delivers Little (2020)
#20It's a bit of an old post, and yes Pipenv is not the go-to tool anymore. pip-tools is okay for people that really, really love their requirements.txt; otherwise we tend to go with Poetry at work. Any folks having a good experience with PDM https://github.com/pdm-project/pdm ?
After a series of bad experiences with Poetry, I switched the packages I maintain to PDM. Although I have hit a few minor snags, the maintainer and other users in the project's github discussions have never failed to help with a fix, workaround, or advice. It's pleasant to use, and I think the only feature I cared about which was in Poetry but not PDM (a publish-to-pypi command), has proven easier to do with the Twine tool anyway.
My situation is I think unusual, in that I need to use a private pypi repo which requires mTLS for both fetching and publishing. Had it not been for that I suspect I'd still be using Poetry, but given the experiences I've had with PDM I wouldn't switch back even if the situation with my repo changed.