Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

161–170 of 293 posts

Re: How Python virtual environments work

#161

Post by lucb1e »

hot_gril wrote: post #155
lucb1e wrote: post #150

Earlier quoted context omitted.

> The cost of setting up, maintaining the environment and onboarding people is just not worth it. I have yet to come across a situation where I need a virtual environment at all. A lot of projects use it, but then lazy me just runs git clone && python3 clone/main.py and it works just fine, sometimes after an apt install python3-somedependency or two. It always seemed weird to me to depend on such a specific version (…

Do you deal with the scientific libs? I remember that whole MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird requirements on dependencies, with Py2 vs 3 confusion added to the mix.

I've used matplotlib and pandas fairly recently because applicants used it in their code, don't remember any specific problems with that. Well, with the dependency installation that is. The applicant loading all data into RAM so that Pandas can operate on it was an issue. (The whole point was that the data was too big to reasonably load into RAM, and would constantly grow as more sensor readings come in, and the questions are such that you can easily stream it and keep a few KB of state in a dictionary to find the answers needed... alas.)

I do remember python2-only being a problem back in the day, but this was solved... hmm, maybe in 2017 somewhen? At least for the packages I used then that had py3 issues before, like sklearn, numpy, and scapy come to mind. I think it more or less coincided with Debian deciding the next release was not going to ship Python 2. Somehow that made everyone 2to3, fix a few remaining bugs, release, done. I'm too young (I'm 30) to really have done much with Python 2 so I didn't have this legacy in my own code (besides a few early files), I've ~always just tried to find and install Python 3 versions of everything.

Re: How Python virtual environments work

#162

Post by crabbone »

groestl wrote: post #50
tmpz22 wrote: post #44

Earlier quoted context omitted.

* Local state is changing such as brew updates or new dependencies are added. * External state is changing such as project contributions So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of ins…

> Ironically using Docker I'd even dare to say that Docker _is_ the answer to the Python's packaging problems, and might have never taken off without that "killer app" that is the Python packaging sh*tshow.

Docker is not the answer to any packaging problems because it's not a packaging tool. I have no idea how people don't understand this... oh wait! I'm talking to Python programmers!

But... I'm not going to hold you in the dark: the reason and the major drive to have a packaging system is that you can define dependencies between packages s.t. users installing package can coordinate and install the stuff they need. Docker simply doesn't do that. You get images. They are completely independent. Whether any two images will work with each other is anyone's guess.

Re: How Python virtual environments work

#163

Post by hot_gril »

neuronexmachina wrote: post #154
hot_gril wrote: post #145

Earlier quoted context omitted.

I've never used anything but vanilla Python venvs, and no they don't work reliably. What does is a Docker container. I keep hearing excuses for it, but the prevalence of Dockerfiles in GitHub Python projects says it all. This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.

> This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things. There's also NodeJS's ability for dependencies to simultaneously use conflicting sub-dependencies.

Yeah, you can't have two deps use different versions of the same sub-dep, cause they flatten everything instead of having a tree. In practice I rarely have issues with this except in React-Native, where it's a common problem, but then again RN is doing some crazy stuff to begin with. Often just force install deps and things work anyway.

Side note, there are way too many React/React-Native "router" type packages, and at least one of them breaks its entire API every update (I think https://reactrouter.com/en/main/upgrading/v5, how are they on version 6 of this). It's so bad that you can't even Google things anymore cause of the naming conflicts.

Re: How Python virtual environments work

#164

Post by [deleted] »

crabbone wrote: post #147
jstx1 wrote: post #66

Earlier quoted context omitted.

With plain venv it’s hard to maintain multiple different Python versions on the same machine; conda makes this much easier. Also on M1/M2 Macs some libraries (especially for ML) are only available through conda-forge.

> With plain venv it’s hard to maintain multiple different Python versions on the same machine; plain venv never advertised itself as a solution to this problem... I don't like this tool, but, sorry to say so, you are barking on the wrong tree. Also, it's not hard to maintain different versions of Python on the same machine without conda. I can literally do this with my eyes closed w/o touching the mouse: it boils do…

[deleted]

Re: How Python virtual environments work

#165

Post by analog31 »

hot_gril wrote: post #155
lucb1e wrote: post #150

Earlier quoted context omitted.

> The cost of setting up, maintaining the environment and onboarding people is just not worth it. I have yet to come across a situation where I need a virtual environment at all. A lot of projects use it, but then lazy me just runs git clone && python3 clone/main.py and it works just fine, sometimes after an apt install python3-somedependency or two. It always seemed weird to me to depend on such a specific version (…

Do you deal with the scientific libs? I remember that whole MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird requirements on dependencies, with Py2 vs 3 confusion added to the mix.

I think the problems have been resolved. For my last few installations of Python, I've just used pip install for those packages, without any issues. Linux and Windows, can't comment about Mac. And the important libraries are all in Py3 now.

I haven't tried out virtual environments yet.

Re: How Python virtual environments work

#166

Post by buildbot »

Filligree wrote: post #159
zzzeek wrote: post #135

Earlier quoted context omitted.

so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2. the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ with a few dozen distros. is the problem solved yet ?

You can't install the CUDA base libraries with pip. You can with Conda. So no, it isn't solved yet.

Good? I don't want my package manager messing with my cuda toolkit setup!

Re: How Python virtual environments work

#167

Post by akprasad »

crabbone wrote: post #129
whalesalad wrote: post #104

Earlier quoted context omitted.

I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.

Lol. You put "simple" and "requirements.txt" unironically next to each other... I mean, I think you genuinely believe that what you suggest is simple... so, I won't pretend to not understand how you might think that. I'll explain: There's simplicity in performing and simplicity of understanding the process. It's simple to make more humans, it's very hard to understand how humans work. When you think about using pip w…

What are specific problems you have with pip + requirements.txt, and why do you believe storing links to wheels is more reliable? Your comment makes your conclusion clear, but I don't follow your argument.

Re: How Python virtual environments work

#168

Post by hot_gril »

analog31 wrote: post #165
hot_gril wrote: post #155

Earlier quoted context omitted.

Do you deal with the scientific libs? I remember that whole MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird requirements on dependencies, with Py2 vs 3 confusion added to the mix.

I think the problems have been resolved. For my last few installations of Python, I've just used pip install for those packages, without any issues. Linux and Windows, can't comment about Mac. And the important libraries are all in Py3 now. I haven't tried out virtual environments yet.

Yeah I'm willing to bet it's gotten easier lately, perhaps because things have settled down. Then again Mac is always its own special case with Py libs.

Re: How Python virtual environments work

#169

Post by tryauuum »

buildbot wrote: post #166
Filligree wrote: post #159

Earlier quoted context omitted.

You can't install the CUDA base libraries with pip. You can with Conda. So no, it isn't solved yet.

Good? I don't want my package manager messing with my cuda toolkit setup!

? what if you need two different versions of cuda toolkit on same machine?

Re: How Python virtual environments work

#170

Post by Nadya »

hot_gril wrote: post #149
danielvaughn wrote: post #43

Earlier quoted context omitted.

I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.

I can't think of any package/dep systems I actually like other than npm. And they're even starting to screw that up with the `import` weirdness instead of the `require` that's been so simple and easy. Rust's system is probably the next best. ObjC/Swift packaging is a flaming disaster in practice, unless it's improved since I jumped that ship. Last time, I remember every single project having to rely on Cocoapods.

Weirdness? `require` was Node weirdness because Javascript lacked imports/exports at the time. The ES6 syntax is remarkably better, allows imports to be async, and doesn't need to run the code to see what should be assigned to `module.exports` allowing it to be statically analyzed which allows tree-shaking. Node's CJS syntax will only work for Node requiring that you transpile and bundle it for browsers. The ES6 syntax will work for Node and the browser.

I see anyone sticking with CJS syntax the same way I see Python devs who continue writing 2.7 code by choice in new projects and not because they are maintaining older projects.

Post reply on HN