I agree with everyone here that Python is a wonderful tool and it is frustrating when everything does not work ootb.
Python environment setup seems complicated and unsolvable
21–30 of 34 posts
Re: Python environment setup seems complicated and unsolvable
#22I've settled on miniconda for virtual environment management and poetry for dependency management. I'm happier with this setup than anything I've tried before.
Re: Python environment setup seems complicated and unsolvable
#23Python packages are mostly fine if they're pure Python, the issue comes when you're installing native C/C++/Fortran dependencies, which most ML and scientific codes have. People who complain about Python vs Go or Rust or Node in this area I think are not comparing like for like at all.
In academia, everyone pretty much uses Conda to get around these issues, because packages there if you stick to the main channel 'just work' but the licensing is prohibitive elsewhere. Many years ago, you couldn't generally expect to install a Python package on your machine if it had compiled dependencies without installing those libraries yourself with your system package manager and having a local compiler toolchain. That's changed because PyPi introduced wheels which allow you to bundled shared objects, and is why Python is now usable on Windows, but it's not perfect as the this post shows. Wheels generally are available for the most popular OS/architecture combinations - x86 processors, Windows, Mac, and 'manylinux' (an old Linux distribution with low version of glibc to maintain compatibility with newer distros. If you fall outside of that (ARM Mac, want to use a distro with MUSL rather than glibc, etc. etc.) then you basically are on your own and need to compile everything from scratch, and you're going to have a lot of... fun... doing so. In my previous job I worked for a University with a POWER9 GPU cluster, and so we used tools like EasyBuild and Spack to try and manage this, but many packages required manual patches in order to get them working.
Re: Python environment setup seems complicated and unsolvable
#24IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry. [1]: https://github.com/python-poetry/poetry/issues/6409
The whole "Python is how science gets done" meme is one of the dumbest things we've allowed to be foisted on otherwise unaware/unsuspecting users (such as the kinds of academics who end up being the victims of the Python ecosystem shitshow). Who knows how many setbacks in science we've suffered, not to mention billions of dollars of productivity lost, sticking to such an unworthy programming system/language/environme…
Once you start to look at scientific packages in other languages, they have the same issues as Python does, because they start to use scientific libraries written in C and Fortran, as rewriting 50+ years of code is actually really hard.
Re: Python environment setup seems complicated and unsolvable
#25> Instead of creating virtual environments using venv, you can use a more sophisticated tool like Poetry which enables you to manage you environments in a much better manner. It seems pretty emblematic of the shitshow that is Python environment management that the top-rated comment is recommending something I've never heard of. I know pyenv and pyvenv (not the same thing!) and virtualenv and venv (not the same thing!…
Funnily enough, Poetry only just very recently started to properly support PyTorch (you know, which seemingly 90% of Python usecases nowadays require). Like, it would not work across different setups with Arm, Cpu and Gpu machines, which is the point of poetry. For years, youd need to use one of a few hacks that had serious downsides. Our deployments still install pytorch stuff separately in every container. I think…
I’ve created about 50 Python projects in the last 3 years that are in production, not a single one uses PyTorch.
Re: Python environment setup seems complicated and unsolvable
#26In terms of tooling Python is a mess. Poetry is indeed quite good, but it still has issues.
I think what python misses is things like cargo, go and npm (dare I say it..), tools that handle packages and run your app in context, making that things generally just work.
Might be anecdotal, but I have had way less cross platform/binary issues in other languages. For node things are usually handled very well these days, rust just compiles stuff (or great docs with clear instructions).
It’s not just environments, it’s formatting, linting, types.
I think python used to be ahead in usability, in the last decade it has fallen behind a bit.
It’s still a good language to use because it can do so much, but I can’t say the experience is enjoyable.
Re: Python environment setup seems complicated and unsolvable
#27My 2 cents:
Since I use a Arch based Linux distribution, I like to update my (main) Python environment just as frequently: at least once a week. My findings so far:
1. Do not interfere with the system managed packages. I.e. use --user when pip install or make the system python lib dir(s) non-writable for the user
2. If you don't like to switch frequently into another venv, just put the packages that interfere with your main (user) env into separate venvs. These are normally young and intensely developed modules, like e.g. textual, or old non-updated modules, like e.g. ecs, or complicated, cross-language stuff, like e.g. numba, scipy. They will separate themselves out after a while.
3. If nonetheless some local packages interfere with system packages (btw, I hate the --break-system-packages crap), make separate users each with the appropriate environment.
Rinse and repeat.
Re: Python environment setup seems complicated and unsolvable
#28I just gave up on learning Python because of this mess. It's really too bad, because I'd like to write utilities with Python that would run on almost any machine. Oh well. When I was still trying to wrangle this, I had followed this guide on setting up Python "the right way." I wonder what y'all think: https://opensource.com/article/19/5/python-3-default-mac
Looks mostly good, but also has extra elements you don’t really need. Pyenv - managing / installing multiple python versions. Then all you need to start is regular virtual environments ‘python -m venv .venv’ with ‘requirements.txt’, but I also recommend poetry. This is for dependency isolation between projects. Packaging up projects for distribution is a whole other venture though!
Thanks!
Re: Python environment setup seems complicated and unsolvable
#29Earlier quoted context omitted.
Poetry is an all-in-one tool that does dependency management using `pyproject.toml` files and its own `poetry.lock` files, as well as package building, virtualenv management, and also has hooks for entrypoints and scripts. It's rather a useful tool and I'm personally using it for dependency management and packing for all my projects moving forwards, though for venvs I'm using `pyenv` and the `pyenv-virtualenv` plugin…
PEP 582 was rejected, FYI. https://peps.python.org/pep-0582/
Bugger.
Thanks for letting me know (embarrassingly I did load up the PEP page to make sure I remembered the right number but I didn't check its status).
Was hoping that it would make things simpler for smaller projects and newbie developers but the rejection reason is solid.
Re: Python environment setup seems complicated and unsolvable
#30I use python pretty much all day long, and would never use a python environment besides the basics. A Docker is more general, serves the same purpose, less mental overhead etc.
Really, could anyone explain why I should care about python environment tools at all? Because we do fine without them.