Live data from Hacker News

How Python virtual environments work

snarky.ca

141–150 of 293 posts

Re: How Python virtual environments work

#141

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.

Is it "generational knowledge and battlefront experience" or just "getting used to the (shitty) way things have always been" and Stockhold Syndrome?

Re: How Python virtual environments work

#142

Earlier quoted context omitted.

I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager.…

> once or twice have I ever encountered a showstopper dependency resolution issue. Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways! Until quite recently, pip didn't give a rat's ass if the dependencies it installe…

It behaves like a kid you send to the store with a 100 dollars

Re: How Python virtual environments work

#143

Earlier quoted context omitted.

I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager.…

> once or twice have I ever encountered a showstopper dependency resolution issue. Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways! Until quite recently, pip didn't give a rat's ass if the dependencies it installe…

I moved to poetry (ergonomics) and publishing wheels with frozen requirements, at least for apps, here's the plugin I used https://github.com/cloud-custodian/poetry-plugin-freeze .. readme has details, tldr freeze the graph for repeatability regardless of tool, Ala pip install works years later.

Re: How Python virtual environments work

#144
post #77

Earlier quoted context omitted.

If it were really that simple, surely all these other solutions wouldn't exist?

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nic…

> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know).

It started in a way similar to how Active Perl started. An attempt to build a reliable, curated and "batteries included" solution. (But it went downhill very quickly).

So, what were the problems:

* Most importantly, using Python on Windows is a pain (it still is, but less so). Many Python packages rely on native libraries which they don't provide. Even if they ship with some binary blob, that blob is often just a wrapper to the actual library. But, at the time Conda was created, even a binary blob was not an option. If you wanted to install stuff, you had to compile it. And, compiling on MS Windows is a world of pain. Especially, if you compile something for Python. Again, credit where it's due, Python sort of fixed some of these problems... but just a tiny bit. It still sucks a lot. So, conda is more of a MSYS2 really: it's a Linux user-space in Windows (firstly), and Python on top of that. But, it kind of also doesn't want to admit it, so, it still pretends it's Windows... but with a bunch of Linux-ey stuff...

* Secondly, pip was a joke until about a year ago. It was not really a package manager (it still isn't) and it couldn't properly solve dependencies (it sort of does today, but poorly because of source-distributions). Conda, when it worked, was at least somewhat consistent. (But usually, it just hangs). Also, conda is a package manager (however awful). I.e. with pip you can run install once, and then you can run a different install command again in the same environment, and god help you to make sure you have consistent dependencies between packages. Conda, in principle, should always keep your environment consistent. (But it hangs more often than not). Also, there's no such thing as installing from source with conda. If you want to use source: git clone, install conda-build and suffer through non-existent documentation and get help from non-existent community of people building conda packages.

* Conda provides a curated set of packages (a.k.a default channel). Anyone worth their salt ditches that channel the moment they install conda and installs everything from conda-forge (cuz its got more stuffs!) So, they tried curated. Didn't work.

---

In a sense, I think that what happened is that conda was too ambitious of a project, using wrong technology to implement it. It also didn't have good enough planning for the kind of ambition they had. So, every time they came to solve a real problem, they didn't have time, human resources and system resources to solve it well. They've accumulated technical debt at close-to-light speed and ended up being a huge mess nobody knows how to deal with.

Some of the obvious mistakes would be: creating too many releases of conflicting packages. They should've worked on some sort of LTS solution, where they release a set of packages with very permissive dependencies of very few versions that had been proven to work well together. Instead it's very common for conda packages to be very peculiar (and without real need to do so) about the versions of their dependencies.

Conda people didn't build good CI. They often release absolutely broken packages, and only discover it retrospectively from community input. (Tensorflow would be a great example). This creates a lot of problems with automatic updates.

Conda gave in to community pressure and decided to build integration with pip. It doesn't work well, and it's not possible to make it work well, but they added this and a lot of people instantly created dependencies on this feature.

Conda picked a bone with some of the infra tools outside of Python ecosystem. In particular with CMake. This just adds an extra aspect of pain trying to build conda packages / work with native libraries / wrappers. It might seem like a minor thing, but it prevented a lot of people who might otherwise release packages both for conda and PyPI from doing so. Often package that is released to conda is ported by someone who's not the package author. It also means that sometimes the names of packages differ between conda and PyPI for the same package.

----

NB. In terms of amount of commands you need to type when working with conda vs with PyPI tools is not noticeably different. Conda is, perhaps, more organized, but is also more buggy due to trying to integrate with various shells in special ways, and failing quite a bit.

Re: How Python virtual environments work

#145

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I've never used anything but vanilla Python venvs, and no they don't work reliably. What does is a Docker container. I keep hearing excuses for it, but the prevalence of Dockerfiles in GitHub Python projects says it all. This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.

Re: How Python virtual environments work

#146

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

Except when you try to move it, or copy it to a different location. This _almost_ made sense back when it was its own script, but it hasn't made sense for years, and the obstinacy to just sit down and fix this has been bafflingly remarkable.

("why not make everyone install their own venv and run pip install?" because, and here's the part that's going to blow your mind: because they shouldn't have to. The vast majority of packages don't need compilation, you just put the files in the right libs dir, and done. Your import works. Checking this kind of thing into version control, or moving it across disks, etc. etc. should be fine and expected. Python yelling about dependencies that do need to be (re)compile for your os/python combination should be the exception, not the baseline)

Re: How Python virtual environments work

#147
post #66

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

With plain venv it’s hard to maintain multiple different Python versions on the same machine; conda makes this much easier. Also on M1/M2 Macs some libraries (especially for ML) are only available through conda-forge.

> With plain venv it’s hard to maintain multiple different Python versions on the same machine;

plain venv never advertised itself as a solution to this problem... I don't like this tool, but, sorry to say so, you are barking on the wrong tree.

Also, it's not hard to maintain different versions of Python on the same machine without conda. I can literally do this with my eyes closed w/o touching the mouse: it boils down to:

    cd ~/src/cpython
    git checkout 3.8
    git fetch --all
    git reset --hard origin/3.8
    git clean -Xdf
    ./configure --with-ensurepip=install --with-system-ffi=yes
    make
    sudo make altinstall
Sorry. I've done this from memory and I kept one eye open. So, I kinda lied. But not really.

Re: How Python virtual environments work

#148

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

If it were really that simple, surely all these other solutions wouldn't exist?

Idk, I still have no idea why yarn exists when there's npm.

Re: How Python virtual environments work

#149

Earlier quoted context omitted.

It sounds mean to say it, but it's 100% true. I moved away from using python wherever I can. I've had colleagues struggle for days to install well used packages like pandas and numpy in conda.

I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.

I can't think of any package/dep systems I actually like other than npm. And they're even starting to screw that up with the `import` weirdness instead of the `require` that's been so simple and easy.

Rust's system is probably the next best.

ObjC/Swift packaging is a flaming disaster in practice, unless it's improved since I jumped that ship. Last time, I remember every single project having to rely on Cocoapods.

Re: How Python virtual environments work

#150
post #38

It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.

> The cost of setting up, maintaining the environment and onboarding people is just not worth it.

I have yet to come across a situation where I need a virtual environment at all. A lot of projects use it, but then lazy me just runs git clone && python3 clone/main.py and it works just fine, sometimes after an apt install python3-somedependency or two.

It always seemed weird to me to depend on such a specific version ("package foo can't be older than 7 months or newer than 4 months"), how even does one manage to use such obscure features that they were removed and so you need an older-than version?

And then there's the people in this thread (seemingly a majority when judging by top level comment loudness) that have trouble with virtual environments and so add another layer on top to manage that virtual environment thing. The heck. Please explain

Post reply on HN