11–20 of 34 posts
> Instead of creating virtual environments using venv, you can use a more sophisticated tool like Poetry which enables you to manage you environments in a much better manner. It seems pretty emblematic of the shitshow that is Python environment management that the top-rated comment is recommending something I've never heard of. I know pyenv and pyvenv (not the same thing!) and virtualenv and venv (not the same thing!…
For instance, a few months ago I was working on an environment for playing around with some libraries that you’d think would commonly go together. But within the community there are very popular libraries that cannot be used together (at recent enough commits to contain features I want) due to version conflicts.
But as I’m a nix person I obviously try to solve that by going rogue. This was my process before I decided to pack it in and wait for the community to sort their shit out:
1. Pull projects at their latest commit instead. This is the “hope the communities already have their shit sorted out” method. nope, no luck.
2. Write a patch that changes the older requirement. Maybe it is API compatible? LOL no such luck.
3. Write a patch that changes the outdated dependency name to some random name, then add an expression to map the outdated dependency to that name instead. Ugh no, there are version incompatibility with its own dependencies too.
4. Write a patch to update the lib with the outdated dependency to use the new API instead - success! On its own. Now use it together with the other libraries… uh oh.
It has a C++ backend that requires a GLIBC version that is incompatible with that used by the other libraries. It needs this because it needs to be built with a specific CUDA-enabled compiler that is quite outdated now, and if you compile it on a newer version it fails to build because the author hardcoded the compatibility matrix into the codebase with c macros.
So yeah there comes a point where you think “hey I want to try out using some fun tools I’ve heard about” and, before you know it, you’re writing a patch for a dependency that allows you to use a patched version of another dependency that allows you to use a patched version of a dependency that you want to add to your project alongside something else, and at some point you’ve just got to quit and hope that in a few weeks there will be version parity.
I just gave up on learning Python because of this mess. It's really too bad, because I'd like to write utilities with Python that would run on almost any machine. Oh well. When I was still trying to wrangle this, I had followed this guide on setting up Python "the right way." I wonder what y'all think: https://opensource.com/article/19/5/python-3-default-mac
Pyenv - managing / installing multiple python versions.
Then all you need to start is regular virtual environments ‘python -m venv .venv’ with ‘requirements.txt’, but I also recommend poetry. This is for dependency isolation between projects.
Packaging up projects for distribution is a whole other venture though!
IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry. [1]: https://github.com/python-poetry/poetry/issues/6409
> Instead of creating virtual environments using venv, you can use a more sophisticated tool like Poetry which enables you to manage you environments in a much better manner. It seems pretty emblematic of the shitshow that is Python environment management that the top-rated comment is recommending something I've never heard of. I know pyenv and pyvenv (not the same thing!) and virtualenv and venv (not the same thing!…
For years, youd need to use one of a few hacks that had serious downsides. Our deployments still install pytorch stuff separately in every container.
I think this is still one of the top (open) issues on the repo.
Also, for sufficiently sophisticated projects, poetry was known to lead to hour long dependency resolving sessions. For example, if you feel like using different pypy sources (eg internal registry, plus some wheels etc), poetry will try to install every package from every of these source registry meaning creating a lock file takes now hours. Not sure if this was ever resolved, but at my company, poetry is know as the thing that never works.
IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry. [1]: https://github.com/python-poetry/poetry/issues/6409
Sure, it doesn’t lead to the same exact environment on every machine, but that stuff never ever works anyway at least with portry.
I use Nix to manage python environments, and it’s quite revealing. For instance, a few months ago I was working on an environment for playing around with some libraries that you’d think would commonly go together. But within the community there are very popular libraries that cannot be used together (at recent enough commits to contain features I want) due to version conflicts. But as I’m a nix person I obviously try…
Poetry, pipenv, etc are all about building dependency tree solvers into python package management.. so if you have package a that depends on package c >= 2.0 and package b that depends on package c Nix basically says “Python packages are lying when they say they need this specific package version. Ignore that and install whatever version is in nixpkgs”.
Which sucks! There are some badly maintained workarounds (poetry2nix and pip2nix come to mind… neither of which works for m1 macs), but the whole stance is just wrong.
I use Nix to manage python environments, and it’s quite revealing. For instance, a few months ago I was working on an environment for playing around with some libraries that you’d think would commonly go together. But within the community there are very popular libraries that cannot be used together (at recent enough commits to contain features I want) due to version conflicts. But as I’m a nix person I obviously try…
Part of the problem is that nix takes an opposite stance from where python seems to be heading… Poetry, pipenv, etc are all about building dependency tree solvers into python package management.. so if you have package a that depends on package c >= 2.0 and package b that depends on package c Nix basically says “Python packages are lying when they say they need this specific package version. Ignore that and install w…
I’m also not surprised that there are still some growing pains with the M1 architecture changeover–when Apple switched from PowerPC to Intel there were software stragglers for a good while.
IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry. [1]: https://github.com/python-poetry/poetry/issues/6409