Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

111–120 of 293 posts

Re: How Python virtual environments work

#111

Post by nose-wuzzy-pad »

cmcconomy wrote: post #17

My personal approach is: - use miniconda ONLY to create a folder structure to store packages and to specify a version of python (3.10 for example) - use jazzband/pip-tools' "pip-compile" to create a frozen/pinned manifest for all my dependencies - use pip install to actually install libraries (keeping things stock standard here) - wrap all the above in a Makefile so I am spared remembering all the esoteric commands I…

This seems simplistic and low drag. Do you have an example you can share?

Thanks!

Re: How Python virtual environments work

#112

Post by frou_dh »

josteink wrote: post #90

All other languages: use whatever packages you like. You’ll be fine. Python: we’re going to force all packages from all projects and repos to be installed in a shared global environment, but since nobody actually wants that we will allow you to circumvent that by creating “virtual” environments you can maintain and have to deal with instead. Also remember to activate it before starting your editor or else lulz. And d…

Python is over 30 years old so it's hardly surprising that it's got plenty baggage in at least some areas.

Re: How Python virtual environments work

#113

Post by ggregoire »

sakex wrote: post #38

It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.

Experienced Python devs run their projects in docker, which solves the 3 issues you listed at the end.

Re: How Python virtual environments work

#114

Post by throwawaymaths »

danielvaughn wrote: post #43
birdstheword5 wrote: post #24

Earlier quoted context omitted.

It sounds mean to say it, but it's 100% true. I moved away from using python wherever I can. I've had colleagues struggle for days to install well used packages like pandas and numpy in conda.

I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.

Elixir's packaging system is quite good. We went from empty project to working stable diffusion in 2h. 1.75 of those hours was installing CUDA.

Re: How Python virtual environments work

#115

Post by ChancyChance »

danielvaughn wrote: post #43
birdstheword5 wrote: post #24

Earlier quoted context omitted.

It sounds mean to say it, but it's 100% true. I moved away from using python wherever I can. I've had colleagues struggle for days to install well used packages like pandas and numpy in conda.

I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.

Same here, I've been using yarn for years, and when I started using venv, I didn't understand why it had to be so complex. Even after reading this article, I still don't see why it is so complex! Yarn/npm has the right idea: dependencies go in the working folder and expect that hierarchy/protocol. Problem solved. The only problem I have with yarn/npm is the problem any package manager has and that is the attrition of dependencies and how to rank their security risk.

Re: How Python virtual environments work

#116

Post by pletnes »

zzzeek wrote: post #77
bobbylarrybobby wrote: post #64

Earlier quoted context omitted.

If it were really that simple, surely all these other solutions wouldn't exist?

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nic…

Before conda, you needed a C and a fortran compiler, a BLAS and LAPACK installation, and various build tools, to install scipy. Scipy is one - 1 - dependency used in science and engineering. The pain used to be massive. Now, we’re complaining about the existence of 2 high quality (not perfect) alternatives, and pip (part of the official python distro) can install tensorflow etc, GPU libs included, with a oneliner.

Re: How Python virtual environments work

#117

Post by jayvius »

zzzeek wrote: post #77
bobbylarrybobby wrote: post #64

Earlier quoted context omitted.

If it were really that simple, surely all these other solutions wouldn't exist?

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nic…

> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know).

Essentially, yes, and justifiably so. Try installing science-y Python packages on Windows written in C. When conda was created in 2012 this meant installing Visual Studio 2005 (for Python 2.7) which was hard to find on Microsoft's own website even back then.

Re: How Python virtual environments work

#118

Post by Jackevansevo »

_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

Likewise, I think people have a negative first experience because it doesn't work exactly like node, throw their toys out the pram and complain on HN for the rest of time.

Guess in taking this stance we're both part of the problem... \s

Re: How Python virtual environments work

#119

Post by wheelerof4te »

pbronez wrote: post #80
buildbot wrote: post #63

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

Until you want to use anything with a c extension..

Why would you want to use Python with a C extension?

If you need performance, just use native code.

Re: How Python virtual environments work

#120

Post by crabbone »

_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

The most important part about venv is that you shouldn't need it. The very fact that it exists is a problem. It is a wrong fix to a problem that was left unfixed because of it.

The problem is fundamental in Python in that its runtime doesn't have a concept of a program or a library or a module (not to be confused with Python's modules, which is a special built-in type) etc. The only thing that exists in Python is a "Python system", i.e. an installation of Python with some packages.

Python systems aren't built to be shared between programs (especially so because it's undefined what a program is in Python), but, by any plausible definition of a program, venv doesn't help to solve the problem. This is also amplified by a bunch of tools that simply ignore venvs existence.

Here are some obvious problems venv doesn't even pretend to solve:

* A Python native module linking with shared objects outside of Python's lib subtree. Most comically, you can accidentally link a python module in one installation of Python with Python from a "wrong" location (and also a wrong version). And then wonder how it works on your computer in your virtual environment, but not on the server.

* venvs provides no compile-time isolation. If you are building native Python modules, you are going to use system-wide installed headers, and pray that your system headers are compatible with the version of Python that's going to load your native modules.

* venv doesn't address PYTHONPATH or any "tricks" various popular libraries (s.a. pytest and setuptools) like to play with the path where Python searches for loadable code. So much so that people using these tools often use them contrary to how they should be used (probably in most cases that's what happens). Ironically, often even the authors of the tools don't understand the adverse effects of how the majority is using their tools in combination with venv.

* It's become a fashion to use venv when distributing Python programs (eg. there are tools that help you build DEB or RPM packages that rely on venv) and of course, a lot of bad things happen because of that. But, really, like I said before: it's not because of venv, it's because venv is the wrong fix for the actual problem. The problem nobody in Python community is bold enough to address.

Post reply on HN