Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

131–140 of 293 posts

Re: How Python virtual environments work

#131

Post by crabbone »

aflag wrote: post #73
_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppo…

I hate PyPI probably even more than you do, but venv doesn't do that. All it does is write a handful of files and make a bunch of symlinks. It doesn't deal with installation of packages.

Re: How Python virtual environments work

#132

Post by antod »

lockhouse wrote: post #45
tmpz22 wrote: post #44

Earlier quoted context omitted.

* Local state is changing such as brew updates or new dependencies are added. * External state is changing such as project contributions So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of ins…

Isn’t it considered best practice not to use brew to install Python for exactly this reason? I’ve always seen it recommended to use pyenv or just download directly from Python.org instead.

> Isn’t it considered best practice not to use brew to install Python for exactly this reason?

The only time I've ever had a big mess with broken Python installs was after using brew on Linux - luckily killing off brew brought the system Python back fine. I'll grudgingly put up with brew on a Mac out of necessity, but keep it away from Linux.

Re: How Python virtual environments work

#133

Post by bobbylarrybobby »

schainks wrote: post #124
bobbylarrybobby wrote: post #64

Earlier quoted context omitted.

If it were really that simple, surely all these other solutions wouldn't exist?

If software were that simple, surely all these other languages wouldn't exist?

Right, software is anything but simple

Re: How Python virtual environments work

#134

Post by crabbone »

whalesalad wrote: post #106
aflag wrote: post #73

Earlier quoted context omitted.

It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppo…

I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager.…

> once or twice have I ever encountered a showstopper dependency resolution issue.

Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways!

Until quite recently, pip didn't give a rat's ass if the dependencies it installed were consistent. It would blink a message in the long stream of vomit it spills on the screen saying something like "you have package X installed of version Y, but package Z wants X of version Q, which will not be installed". And happily streamed more garbage to your screen.

It was an issue that was filed against pip for something like 12 years until it got resolved about a year or so ago. Even after it got resolved a lot of people tried to upgrade, saw that that would "break" their deployment, and rolled back to the latest broken version.

Things are sort of improving gradually since then, but we are light years away from the system working properly, and I know you don't want to know why, but I'll tell you anyways!

So, when for whatever reason pip doesn't find a dependency it thinks you need, a lot of packages, when they roll out their "releases", they upload also what Python calls "source release". Which should have never been treated as an installation option, but it is, and is treated like that by default. So, what will happen once pip finally gives up on finding a match, right, you guessed it! -- It's going to try to build it! Installing build dependencies along the way. What you get in the end is anyone's guess, but most likely, it's something broken because the developers who made this release didn't make a release specifically for your version.

Don't despair. There's a flag you can use with pip install that should prevent it from trying to build stuff. But two bad things will happen to you if you use it: in any non-trivial project your dependencies will irreparably break. And, who knows if that flag is implemented correctly... nobody in the real world is using that. So, who knows, maybe it'll format your hard drive along the way.

Re: How Python virtual environments work

#135

Post by zzzeek »

zzzeek wrote: post #77
bobbylarrybobby wrote: post #64

Earlier quoted context omitted.

If it were really that simple, surely all these other solutions wouldn't exist?

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nic…

so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2.

the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ with a few dozen distros.

is the problem solved yet ?

Re: How Python virtual environments work

#136

Post by Havoc »

spyremeown wrote: post #65
Havoc wrote: post #55

These days I'm just throwing each project into a fresh LXC on a server. All these different languages have their own approach and each then also user/global/multiple versions...it's just not worth figuring out

Question: what makes you choose LXC over Docker?

Much of a sameness really but I prefer the more persistent disk style of lxc plus ssh plus vscode ssh remote extension.

Depends on task though I've got dockers and VMs in use too

Re: How Python virtual environments work

#137

Post by crabbone »

buildbot wrote: post #63

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

I remember when Conda just appeared... I was so high no hopium... well, the word "hopium" didn't exist yet...

Anyways. Today I have to help scientists to deal with it. And... I didn't think it was possible to be worse than pip or other Python tools, but they convinced me it is. Conda is the worst Python program of note that I had ever dealt with. It's so spectacularly bad it's breathtaking. You can almost literally take a random piece of its source code and post it to user boards that make fun of bad code. When I have a bad day, I just open its code in a random file, and like that doctor who was happy running around the imaging room exclaiming "I'm so, so, so happy! I'm so unimaginably happy that I'm not this guy! (pointing at an X-ray in his hand)" I'm happy I'm not the author of this program. I would've just died of shame and depression if I was.

Re: How Python virtual environments work

#138

Post by crabbone »

bobbylarrybobby wrote: post #64
buildbot wrote: post #63

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

If it were really that simple, surely all these other solutions wouldn't exist?

Any program with real world application solves some problem. That's not the point. The point is in how does it do it. And when it comes to conda... it's like MS Outlook on steroids. I cannot really think about a better way to describe it. It's like a movie that can be so bad that it's actually good.

Re: How Python virtual environments work

#139

Post by PaulHoule »

dima55 wrote: post #130
PaulHoule wrote: post #51

Earlier quoted context omitted.

Python has always struggled with distribution maintainers that just don’t get it. Python barely survived becoming the default scripting language on Red Hat and other Linux distros which was a major obstruction to the Python 3 transition. If the new cohort of pandas and scikit-learn users had not been such a force of nature we”d be talking today about Python the way we do about Perl. Not installing venv is a serious c…

What's the crime in Debian? The "right answer about how to manage packages" is "apt install package". Anything else (including venvs) is a hack

The word package is overloaded. There is a Debian package and there is a Python package.

If somebody wanted to package a modern Python application as Debian package they’d have to make a Debian package that contains several Python packages maybe as a venv or executable wheel, it is a solvable problem but a bit like comparing tic tac toe to 12d chess with supersymmetry in terms of the ambition of Linux distros if not the difficulty.

If you installed python packages as debs without any isolation you'd never be able to install two applications that depended on different versions of the same Python package.

The best thing about Java is that it is so xenophobic that nobody expects to have a working Java runtime installed with the OS so you never have to deal with a broken runtime installed with the OS. JAR files are remarkably self-sufficient so "write once one anywhere" is way closer to the truth than it is on "DLL Hell" platforms like Windows and Linux.

Re: How Python virtual environments work

#140

Post by evntdrvn »

Nadya wrote: post #47
danielvaughn wrote: post #43

Earlier quoted context omitted.

I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.

Personally, zero complaints about Cargo (Rust) and very minimal complaints about NuGet (C#/.NET). My issues around NuGet are probably self-created because I refuse to learn the CLI [0] for it and I've had occasional issues with Visual Studio's UI for managing things. https://learn.microsoft.com/en-us/nuget/reference/nuget-exe-...

In a lot of ways, Paket is significantly better than NuGet, if you ever want to try something new :) It uses a lockfile approach like Cargo, has better dependency resolution, etc https://fsprojects.github.io/Paket/index.html
Post reply on HN