Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

151–160 of 293 posts

Re: How Python virtual environments work

#151

Post by anongraddebt »

whalesalad wrote: post #104
_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.

Twice bricking my laptop’s ability to do python development because of venv + symlink bs was the catalyst I needed to go all-in on remote dev environments.

I don’t drive python daily, but my other projects thank Python for that.

Re: How Python virtual environments work

#152

Post by lucb1e »

ggregoire wrote: post #113
sakex wrote: post #38

It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.

Experienced Python devs run their projects in docker, which solves the 3 issues you listed at the end.

Experienced Python dev checking in here. I use neither containers nor virtual environments. Honestly not sure what all the fuss is about.

Maybe everyone has come to think we need layers on layers on layers because management tools (like venv) are blogged and talked about, whereas it's a bit dull to write/talk about nothing (i.e. not using/needing such tools)? I genuinely wonder

Re: How Python virtual environments work

#153

Post by hot_gril »

swyx wrote: post #10
Max_Limelihood wrote: post #8

Answer: they don’t (Seriously, I’ve gotten so fed up with Python package management that I just use CondaPkg.jl, which uses Julia’s package manager to take care of Python packages. It is just so much cleaner and easier to use than anything in Python.)

also there's like 3 different flavors of virtual env now and me being 8 years out of date with my python skillz i have no idea what the current SOTA is with python venv tooling :/ i dont need them demystified, i need someone smarter than me to just tell me what to do lol

> i dont need them demystified, i need someone smarter than me to just tell me what to do lol

Dockerfile ;)

Re: How Python virtual environments work

#154

Post by neuronexmachina »

hot_gril wrote: post #145
_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I've never used anything but vanilla Python venvs, and no they don't work reliably. What does is a Docker container. I keep hearing excuses for it, but the prevalence of Dockerfiles in GitHub Python projects says it all. This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.

> This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.

There's also NodeJS's ability for dependencies to simultaneously use conflicting sub-dependencies.

Re: How Python virtual environments work

#155

Post by hot_gril »

lucb1e wrote: post #150
sakex wrote: post #38

It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.

> The cost of setting up, maintaining the environment and onboarding people is just not worth it. I have yet to come across a situation where I need a virtual environment at all. A lot of projects use it, but then lazy me just runs git clone && python3 clone/main.py and it works just fine, sometimes after an apt install python3-somedependency or two. It always seemed weird to me to depend on such a specific version (…

Do you deal with the scientific libs? I remember that whole MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird requirements on dependencies, with Py2 vs 3 confusion added to the mix.

Re: How Python virtual environments work

#156

Post by hot_gril »

lockhouse wrote: post #45
tmpz22 wrote: post #44

Earlier quoted context omitted.

* Local state is changing such as brew updates or new dependencies are added. * External state is changing such as project contributions So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of ins…

Isn’t it considered best practice not to use brew to install Python for exactly this reason? I’ve always seen it recommended to use pyenv or just download directly from Python.org instead.

I avoid brew whenever possible, partially because it does weird stuff just to avoid installing things as root. MacPorts is nice but doesn't always have the port you want. Something like Python with a convenient .pkg download directly from python.org makes the choice obvious.

That said, some Python packages rely on native code that you might find yourself brew-installing. That can be a nightmare.

Re: How Python virtual environments work

#157

Post by [deleted] »

crabbone wrote: post #147
jstx1 wrote: post #66

Earlier quoted context omitted.

With plain venv it’s hard to maintain multiple different Python versions on the same machine; conda makes this much easier. Also on M1/M2 Macs some libraries (especially for ML) are only available through conda-forge.

> With plain venv it’s hard to maintain multiple different Python versions on the same machine; plain venv never advertised itself as a solution to this problem... I don't like this tool, but, sorry to say so, you are barking on the wrong tree. Also, it's not hard to maintain different versions of Python on the same machine without conda. I can literally do this with my eyes closed w/o touching the mouse: it boils do…

[deleted]

Re: How Python virtual environments work

#158

Post by crabbone »

lockhouse wrote: post #41
sakex wrote: post #38

It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.

Excuse my ignorance, but aren’t Virtual Environments something you setup once per project? Why would that be a dealbreaker? How is it any more difficult than putting everything in a docker container like all the cool kids are doing these days?

> but aren’t Virtual Environments something you setup once per project?

My typical count is 2-3 per project per machine I work on (could be anywhere from one to one hundred). Then there's a different number of people who need to set up these environments on different machines too (and sometimes require my support).

So, the answer is: who knows?

> putting everything in a docker container

I think, you meant image, not container, but that's an easy mistake to make. And the answer is: both options are terrible. It's a choose your poison kind of thing.

> cool kids

My impression from working with the cool kids is that our industry selects for particular taste in t-shirts rather than knowledge or experience. I'm afraid that the correlation might swing into the negative, if we take either knowledge or experience vs coolness.

---

Most importantly: venv is not the problem. It's a bad fix for a problem. Bad as in it doesn't fix the problem actually, it pretends to. I mean, maybe it covers some 90% of the problem -- who knows, I didn't count. So, it kinda works for a lot of people. But, honestly, I'd prefer that the problem was fixed s.t. it doesn't require venv. It's kind of like discussing various dental prosthetics options: it's better to just have healthy teeth.

Re: How Python virtual environments work

#159

Post by Filligree »

zzzeek wrote: post #135
zzzeek wrote: post #77

Earlier quoted context omitted.

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nic…

so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2. the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ with a few dozen distros. is the problem solved yet ?

You can't install the CUDA base libraries with pip. You can with Conda.

So no, it isn't solved yet.

Re: How Python virtual environments work

#160

Post by hot_gril »

groestl wrote: post #50
tmpz22 wrote: post #44

Earlier quoted context omitted.

* Local state is changing such as brew updates or new dependencies are added. * External state is changing such as project contributions So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of ins…

> Ironically using Docker I'd even dare to say that Docker _is_ the answer to the Python's packaging problems, and might have never taken off without that "killer app" that is the Python packaging sh*tshow.

Yep. Any big Python repo is gonna have a Dockerfile. If something doesn't fix itself, users take matters into their own hands.
Post reply on HN