I haven't used these since docker
How Python virtual environments work
11–20 of 293 posts
Re: How Python virtual environments work
#12This writeup needs work. > So while you could install everything into the same directory as your own code (which you did, and thus didn't use src directory layouts for simplicity), there wasn't a way to install different wheels for each Python interpreter you had on your machine so you could have multiple environments per project (I'm glossing over the fact that back in my the day you also didn't have wheels or edita…
I expect most everyday python users know what these things are. I also expect this was targeted at python users who use these things but haven't thought deeply about them.
Charitably, I will assume you are a non python user, and that's why this is a miss for you.
Re: How Python virtual environments work
#13Answer: they don’t (Seriously, I’ve gotten so fed up with Python package management that I just use CondaPkg.jl, which uses Julia’s package manager to take care of Python packages. It is just so much cleaner and easier to use than anything in Python.)
also there's like 3 different flavors of virtual env now and me being 8 years out of date with my python skillz i have no idea what the current SOTA is with python venv tooling :/ i dont need them demystified, i need someone smarter than me to just tell me what to do lol
The reality is that if you ask 3 different people you're going to get 3 different answers. They're fundamentally the same, just a matter of package management. As far as I'm aware, the current "SOTA" is Poetry. I liked Pipenv for quite some time, but Poetry is just so much faster IME.
Re: How Python virtual environments work
#14I haven't used these since docker
With docker, do you use debugging in pycharm/vscode, or just for compiling/shipping?
Re: How Python virtual environments work
#15Answer: they don’t (Seriously, I’ve gotten so fed up with Python package management that I just use CondaPkg.jl, which uses Julia’s package manager to take care of Python packages. It is just so much cleaner and easier to use than anything in Python.)
I hate python package management - I really do. But I've never actually had a problem with virtual environments, and I think it's because I just use virtualenv directly (rather than conda or whatever else).
I have these aliases in my .bashrc, and I can't remember the last time I had a major issue.
alias venv='rm -rf ./venv && virtualenv venv && source ./venv/bin/activate'
alias vact='source ./venv/bin/activate'
alias pinstall='source ./venv/bin/activate && pip install . && pip install -r ./requirements.txt && pip install ./test_requirements.txt'
I don't have all the fancy features, like automatically activating the virtualenv when I cd into the directory, but I've always found those to be a bigger headache than they are worth. And if I ever run into some incompatibility or duplicate library or something, I blow away the old venv and start fresh. It's a good excuse to get up and make a cup of tea.
Re: How Python virtual environments work
#16I haven't used these since docker
That's just giving up.
Re: How Python virtual environments work
#17My personal approach is:
- use miniconda ONLY to create a folder structure to store packages and to specify a version of python (3.10 for example)
- use jazzband/pip-tools' "pip-compile" to create a frozen/pinned manifest for all my dependencies
- use pip install to actually install libraries (keeping things stock standard here)
- wrap all the above in a Makefile so I am spared remembering all the esoteric commands I need to pull this all together
in practice, this means once I have a project together I am:
- activating a conda environment
- occasionally using 'make update' from to invoke pip-compile (adding new libraries or upgrading), and
- otherwise using 'make install' to install a known working dependency list.
Re: How Python virtual environments work
#18> One point I would like to make is how virtual environments are designed to be disposable and not relocatable. Is the author saying that relocating them will actually break things, or that it's just as easy to recreate them in a different location? Because I've moved my venv directories and everything still seemed to work OK. Did I just get lucky?
There’s also relocating across machines. For example, maybe your build environment has access to internal registries but your release environment does not. I naively thought you could build your venv and just copy to the new machine (both environments were Ubuntu) but ran into errors (due to links breaking). We also used pex for a bit, which is kind of like building a binary of a venv, and that eventually stopped working too when the C ABI was no longer the same between environments. There didn’t seem to be an easy way to pick the ABI version to target when creating the pex file, so I gave up and just downloaded the wheels for internal packages in the build.
Re: How Python virtual environments work
#19> One point I would like to make is how virtual environments are designed to be disposable and not relocatable. Is the author saying that relocating them will actually break things, or that it's just as easy to recreate them in a different location? Because I've moved my venv directories and everything still seemed to work OK. Did I just get lucky?
I’ve tried to move things and broken everything. (Conda environments). I tried replacing the paths in the files and it didn’t work. We run a bunch of different tools with various python requirements and would like to be able to duplicate them for the next tool.
We ended up making a new environments for each. Honestly it’s a bit of a mess.