Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don't really need a virtualenv

frostming.com

Post reply on HN
Print view View original

141–148 of 148 posts

Re: You don't really need a virtualenv

#141

Post by mejutoco »

Dryondristica wrote: post #130
Traubenfuchs wrote: post #43

Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works with maven, which is working so damn well it last got updated in November 2019, and the biggest fight going is whether you should use gradle as client instead of maven to access the same dependency ecosystem? I had to set up Python projects for some machine lea…

People always criticise Python for dependency management, but I've never found it to be that bad. virtualenv with requirements.txt has been completely fine in my experience with medium-sized projects. If necessary you can pin versions with a separate version-locked requirements.txt, and populate that from a base requirements file.

Came here to say exactly this. This has been my experience as well

Re: You don't really need a virtualenv

#142

Post by curryst »

isignal wrote: post #118
BerislavLopac wrote: post #107

Earlier quoted context omitted.

Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it. On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from…

All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make). The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.

I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller " and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test.

> All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users.

virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.

Re: You don't really need a virtualenv

#143

Post by curryst »

budafish wrote: post #36
darkteflon wrote: post #4

This was a good article and looks like an interesting project. Regardless of whether or not you choose to _deploy_ with Docker, _developing_ in Docker containers using the VS Code Remote extensions really solved all of Python's (and Javascript's) annoying packaging problems for me, with the bonus that you also get to specify any additional (non-Python) dependencies right there in the repo Dockerfile and have the deve…

Ive never quite managed to get the hang of setting up a dev container. I'ts a bit too tricky for me. I have resorted to using a throwaway dev container which mimics the local storage of packages. Hopefully someone find the below bash function useful. The only issue I have using this method is that I cant use VSCode to Debug, which I am hoping to find a nice solution for, but nothing yet. function python() { docker ru…

Couldn't you use this? https://code.visualstudio.com/docs/remote/containers

Re: You don't really need a virtualenv

#144

Post by curryst »

sethammons wrote: post #29
orf wrote: post #26

Earlier quoted context omitted.

Honestly just use poetry.

This comment would be vastly more valuable (to me at least) if you said why. The parent’s workflow mirrors my own and doesn’t feel onerous. Being completely unfamiliar, their site says: > Poetry either uses your configured virtualenvs or creates its own to always be isolated from your system. So it is just a pretty wrapper? I’m missing the value add.

One value add is a reduced cognitive overhead. I don't have to remember where my virtualenv is; Poetry remembers for me.

The "poetry run" command is extremely nice. You can do something like "poetry run pytest" and put that in a Makefile and be done. You don't have to worry about making sure you're in the virtualenv. I basically don't drop into a full shell anymore (although "poetry shell" lets you do that) because I don't like having to juggle whether I'm in a virtualenv or not.

"poetry build" will build your project into tarballs that pip can install. It's not earth-shattering, but it's nice to have it one tool.

The value add is a large number of QoL improvements. I don't think there are any features that other solutions can't compete on; Poetry just makes it easier to do the same things. I would highly suggest it for anything where other people are going to work on it, because of the QoL, but for personal projects I don't think it really matters.

Re: You don't really need a virtualenv

#145

Post by wiredfool »

wiredfool wrote: post #124
nicoburns wrote: post #109

Earlier quoted context omitted.

Ah I see, that makes sense. I believe Node modules typically fall back to building from source in those cases. Which can be a little slow, but generally works reliably.

I can see that, but we rely on a bunch of system libraries (libjpeg/zlib, + optional libtiff/freetype/etc, and python headers as well). Pretty much by definition, if we don't provide binaries, it's not going to be that simple. I'd say maybe 25-50% of developer linux workstations have the appropriate packages and will compile out of the box, and if they don't, it's almost always a one line command to install them with…

Following up, two other things we don't do are 3) Fortran dependencies and 4) Rust.

Re: You don't really need a virtualenv

#146

Post by danmur »

user5994461 wrote: post #140
danmur wrote: post #121

Earlier quoted context omitted.

It's absolutely not required, I assure you. If you run the Python binary inside the venv you have the same environment as if you'd activated the venv, but only for that execution and not affecting your shell. Try making a venv, running python, and looking at sys.path.

I assure you 100%, it is required for everything to work, it's not just here for the show. One of the thing that the virtual environment does is modify the PATH to lookup executables from the virtual environment first. This is to ensure that when a script calls binaries like mypy/python, it's going to find the same binary from the environment it's currently running in. Typically if the environment is not activated, p…

I see we will not agree, and this thread is mostly dead anyway, but for posterity: it sounds like you haven't installed mypy into the venv you're using. I can see how that would not work. I don't try and reference python scripts that I haven't installed using the venv so perhaps that's why I've never encountered this situation. If it's part of my project it's installed into the venv (and if it's a dev dependency like mypy, which I do use, it gets installed as a dev dependency so it doesn't end up in releases).

Re: You don't really need a virtualenv

#147

Post by erfgh »

rk06 wrote: post #79
erfgh wrote: post #38

Earlier quoted context omitted.

What do you mean "python install[s] globally and only globally"? That's not true. You can instruct pip to install to any place you want.

the thing is python cli has no concept of locally scoped libraries. you can install libs wherever you want, but all python.exe will refer to the same global location. this is why you need to activate virtualenv every time you open a new command line, so that it can set ENV variables for the current session and force python to use the locally scoped folders. While in C# et all. the compiler knows to check files locall…

You don't need a virtualenv. You can simply launch python with the PYTHONPATH environment variable (there are also other ways).

Re: You don't really need a virtualenv

#148

Post by budafish »

curryst wrote: post #143
budafish wrote: post #36

Earlier quoted context omitted.

Ive never quite managed to get the hang of setting up a dev container. I'ts a bit too tricky for me. I have resorted to using a throwaway dev container which mimics the local storage of packages. Hopefully someone find the below bash function useful. The only issue I have using this method is that I cant use VSCode to Debug, which I am hoping to find a nice solution for, but nothing yet. function python() { docker ru…

Couldn't you use this? https://code.visualstudio.com/docs/remote/containers

That's my point. I've always found it hard to set up dev containers. Also each dev container requires its own container, so I may have 10 python docker containers. My implementation only requires 1 image and temporary containers.
Post reply on HN