Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

281–290 of 293 posts

Re: How Python virtual environments work

#281

Post by MrJohz »

bombolo wrote: post #258
MrJohz wrote: post #244

Earlier quoted context omitted.

I mean, a project needs regular care and maintenance, however you organise it. If you're never scheduling time to maintain your dependencies, you're going to be in trouble either way. But at least if you lock your dependencies, you know what will actually get installed, and you can find the buggy or insecure versions. We found a bug on a Python project I worked on recently that only seemed to happen on certain machin…

You should really start using linux distributions. These problems are all solved and have been solved for a long time.

That's definitely a solution, but it comes with its own problems, in particular that you add a significant dependency on what is essentially a middleman organisation trying to manage all possible dependencies. This doesn't scale very well, particularly because there's a kind of M×N problem where M packages can each have N versions which can be depended on. In practice, most distros tend to only support one version of each package, which makes the job easier for the distro maintainer, but makes things harder for everyone else (library authors get bug reports for problems they've already fixed, end users have less ability to choose the versions they need, etc).

In particular, it also makes upgrading a much more complex task. For example, React releases new major versions on a semi regular basis, each one containing some breaking changes, but not many. Ideally there wouldn't be any, but breaking changes are inevitable with any tool as situations change and the problem space becomes better understood. But because the NPM ecosystem generally uses locked dependency lists, end users can upgrade at their leisure, either with small changes every so often, or only upgrading when there's a good reason to do so. Both sides can be fairly flexible in how they do things without worrying about breaking something accidentally.

Under a Linux distribution model however, those incremental breaking changes become essentially impossible. But that means that either projects accumulate cruft that can't ever be removed and makes maintainers' and users' lives more complex, or projects have to do occasional "break everything" releases a là Python 2/3 in order to regain order, which is also more work for everyone. There is a lot less flexibility on offer here.

I don't think these sorts of problems disqualify the Linux distribution model entirely - it does do a lot of things well, particularly when it comes to security and long-term care. But there's a set of tradeoffs at play here, and personally I'd rather accept more responsibility for the dependencies that I use, in exchange for having more flexibility in how I use them. And given the popularity of language-specific package repositories that work this way, I get the feeling that this is a pretty common sentiment.

Re: How Python virtual environments work

#282

Post by selcuka »

lmm wrote: post #227
selcuka wrote: post #224

Earlier quoted context omitted.

I have multiple Python versions on both my Macbook M1 and my Linux laptop, with several dozen venvs. I honestly don't see why others are having so many issues. Probably bad documentation/tutorials.

Presumably they're not managed by your system package manager? Or you have one of the rare and relatively new distributions (certainly not typical Linux) that can handle that smoothly, like NixOS?

Quite typical: Linux Mint, managed by apt with a third party repository [1].

On Mac I simply use the official installers from python.org.

[1] https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa

Re: How Python virtual environments work

#283

Post by bombolo »

JohnFen wrote: post #269
bombolo wrote: post #238

Earlier quoted context omitted.

why? Also, you don't have to use them.

It strikes me that virtual environments are a fairly ugly hack to smooth over the fact that Python is not a stable language. It changes a lot, requiring the use of particular runtimes for particular Python code, requiring the installation of multiple runtimes. That's a pretty serious downside to the language. Virtual environments are needed to help people deal with that downside.

Uh? To me they're just a convenient quick way to install stuff that I don't want to install system-wide, if I want to do some quick experimenting.

The normal, permanent, stuff gets installed system wide the normal way, with apt.

Re: How Python virtual environments work

#284

Post by crabbone »

groestl wrote: post #280
crabbone wrote: post #278

Earlier quoted context omitted.

> Once. On a build server. I didn't ask where, I asked how . How will you know what packages do you need to install? Are you going to go on pypi.org, download them one by one, unzip, look for requirements, then go to pypi.org again, download more, and so on, until you have enough packages and have figured out all the requirements? If so, I have very bad news for you: your single installation might take weeks, possibl…

You have a very abrasive style of dialog, combined with an unwillingness to consider alternative POVs in your discussion, bordering on trolling. I like discussion, but I don't enjoy this. With that, I will disengage now.

You don't have an alternative point of view. You don't have a point of view, because, again, you don't understand something very basic about this subject.

Re: How Python virtual environments work

#285

Post by Filligree »

buildbot wrote: post #192
tryauuum wrote: post #169

Earlier quoted context omitted.

? what if you need two different versions of cuda toolkit on same machine?

I may be wrong, but I don’t think you would want that - it should match with your driver version. Weird stuff happens with minor version mismatches

Probably true, but it usually works anyway. And a lot of ML code depends on exact versions of the CUDA libraries, so you don't get a choice.

Re: How Python virtual environments work

#286

Post by oblvious-earth »

MrJohz wrote: post #234
oblvious-earth wrote: post #220

Earlier quoted context omitted.

The simple solution to this with pip is constraints file: pip install pip freeze > constraints.txt And now in any new environment: pip install -c constraints.txt Now you can install prod requirements or dev requirements or whatever other combination of requirements you have and you guarantee to have the exact same subset of packages, no matter what your transitive dependencies are doing. You can use pip-compile from…

This is true, but now you're explicitly depending on all of your transitive dependencies, which makes updating the project a lot harder. For example, if a dependency stops pulling in a transitive dependency past a certain version, you'll need to either recreate the constraints file by reinstalling everything, or manually remove the dependencies you don't need any more. Also pip freeze does not emit a constraints file…

> This is true, but now you're explicitly depending on all of your transitive dependencies

They're constraints not dependencies they don't need to be installed and you can just update your requirements as you need and regenerate them.

> Also pip freeze does not emit a constraints file, it emits (mostly) a requirements file. This distinction is rarely important, but when it is, it can cause a lot of problems with this workflow. For example, a constraints file cannot include any information about which extras are installed, which pip freeze does by default

Pip freeze does not use extras notation, you just get extra packages listed as individual dependencies. Yes there is an important distinction between constraints and requirements but Pip freeze uses an intersecting subset of the notation.

> You also can't have installed the current project in editable mode if you want the simple "pip freeze" workflow to work correctly

That's why the workflow I gave to generate the constraints didn't use the -e flag, you generate the constraints separately and then can install however you want, editable or not.

> From my experience trying basically every single option in the packaging ecosystem, there aren't really any solutions here. Even Poetry, which is pretty much best-in-class for actually managing dependencies, struggles with workspace-like installations and more complicated build scripts. Which is why I think pretty much every project seems to have its own, subtly unique build/dependency system.

People have subtly different use cases that make a big impact on what option is best for them. But I've never been able to fit Poetry into any of my use cases completely, whereas a small shell script to generate constraints automatically out of my requirements has worked exceedingly well for pretty much every use case I've encountered.

Re: How Python virtual environments work

#287

Post by Doxin »

Wowfunhappy wrote: post #259
Doxin wrote: post #248

Earlier quoted context omitted.

When creating the venv it hardcodes some paths so the python interpreter knows where to find its modules and the likes. That said, re-creating a venv shouldn't be hard and if it is you're doing something wrong in your development setup.

What am I doing wrong? As far as I can tell, I have to: 1. Copy my code out from the venv folder 2. Delete the venv folder 3. Make a new venv 4. Copy my code back into the new venv folder 5. Re-install dependencies This doesn't take much longer than 60 seconds, but that's 55 seconds more than I want to spend. How is this a good process? It just makes me avoid using python (at least when I'd need anything outside the…

Your code should not be inside the venv folder. For reference my projects usually look something like this:

     project
     ├─ venv
     |  ╰─ ...
     ├─ pyproject.toml
     ╰─ project
        ├─ __init__.py
        ├─ __main__.py
        ╰─ app.py
Which means recreating the venv is as easy at removing the venv folder, creating a new venv, and running `pip install -e .` when using pyproject.toml or `pip install -r requirements.tx` when using a requirements file.

This of course doesn't quite solve the moving the folder issue, for which unfortunately there isn't an amazing solution currently. One thing you can do is have the venv somewhere else entirely, That way you can keep the venv in a fixed place so it doesn't break but still move the code to wherever you want to put it. In the use-case for tiny scripts like you do you might be better served not using a venv, and just using `pip install --local` for all your packages. Which is a bit messy but has served me for years and years before I landed on the pattern I use now.

Another "unfortunately" is that none of this stuff is documented very well. Writing a working pyproject.toml for example requires switching between the PEP introducing them, the pip documentation, and the setuptools documentation.

Re: How Python virtual environments work

#288

Post by ElectricalUnion »

mixmastamyk wrote: post #279
ElectricalUnion wrote: post #267

Earlier quoted context omitted.

If a rogue package rm's your root directory as root, you need a bit more that a few shell commands to fix it.

Can't happen unless you install as root. You're NOT DOING THAT are you? Also LiveCDs are a thing for about twenty years. Recovery has never been easier, even after hardware failure.

It doesn't even need root to cause damage most of the time, it just needs to overwrite all files under your user by mistake.

> Recovery has never been easier, even after hardware failure.

If you can use a LiveCD to repair it, it most likely wasn't a hardware failure to start.

Re: How Python virtual environments work

#289

Post by ElectricalUnion »

bombolo wrote: post #275
ElectricalUnion wrote: post #264

Earlier quoted context omitted.

What happens when your distribution only have old versions, or worse, no versions of the libraries you need? You hoop distribution? You layer another distribution like Nix or Anaconda over your base distribution? You give up and bundle another entire distribution in a container image?

You make a package for the thing you need.

So the "solution to packages" is to make your own package with someone's else package?

If it's that simple, how come no one already did all that work for us?

Re: How Python virtual environments work

#290

Post by pnt12 »

peterhil wrote: post #127
tomalaci wrote: post #49

I would highly recommend Poetry for python package management. It basically wraps around pip and venvs offering a lot of convenience features (managing packages, do dist builds, etc.). It also works pretty nicely with Tox. I would recommend using virualenvs.in-project setting so Poetry generates venv in the project folder and not in some temporary user folder.

I just compared and evaluated Hatch, Flit, Poetry and Pdm and found Pdm to be most robust and slimmest. Hatch was a good second option, and Poetry and Hatch are easy to use, but have too much bloat and magic.

When I tried pdm it wasn't stable yet and messed up my paths.

My experience with poetry has been great. I only disliked that they had auto-update on when locking files, but they changed the default.

Post reply on HN