Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

271–280 of 293 posts

Re: How Python virtual environments work

#271

Post by groestl »

crabbone wrote: post #261
groestl wrote: post #232

Earlier quoted context omitted.

Your post reminds me of the saying with the faster horses. Docker doesn't solve Python's packaging problems by being a better packaging system. It solves it by sidestepping the issue of users installing packages themselves, and shipping the disk of the one machine where it once worked. It's a bad linker. (not a fan of the condescending tone in your post, btw)

> It solves it by sidestepping the issue Lol. It doesn't. How are you going to create Docker images? Run more Dockers inside Docker? And then Dockers all the way down? You still need to install the packages... just while creating an image. Makes absolutely no difference whether you do it inside or outside, you still need something that can install and coordinate packages. Docker is simply irrelevant to this problem.

> How are you going to create Docker images?

Once. On a build server. Or a dev's workstation who had the time to fiddle with all the breakage that is python envs. This absolutely does sidestep the issue of distributing Python packages to your users.

We're talking about different problems here. You are talking about library dependencies for devs (and I understand that I'm a bit off topic for this thread)

Re: How Python virtual environments work

#272

Post by ElectricalUnion »

PaulHoule wrote: post #139
dima55 wrote: post #130

Earlier quoted context omitted.

What's the crime in Debian? The "right answer about how to manage packages" is "apt install package". Anything else (including venvs) is a hack

The word package is overloaded. There is a Debian package and there is a Python package. If somebody wanted to package a modern Python application as Debian package they’d have to make a Debian package that contains several Python packages maybe as a venv or executable wheel, it is a solvable problem but a bit like comparing tic tac toe to 12d chess with supersymmetry in terms of the ambition of Linux distros if not…

The funny part about Java is that "Enterprise Java" used to depend a lot more on things being in the base application server, and because that was, as usual, complete hell, it now went the complete other way and microservices will often "just bundle the entire application server" in the uberjar as a result.

Re: How Python virtual environments work

#273

Post by bozobuttz »

oblvious-earth wrote: post #220
johtso wrote: post #214

Earlier quoted context omitted.

Isn't pip + requirements.txt insufficient for repeatable deployments? You need to pin all dependencies not just your immediate project dependencies, unless you want some random downstream update to break your build. I guess you can do that by hand.. but don't you kind of need some kind of a lock file to stay safe/sane?

The simple solution to this with pip is constraints file: pip install pip freeze > constraints.txt And now in any new environment: pip install -c constraints.txt Now you can install prod requirements or dev requirements or whatever other combination of requirements you have and you guarantee to have the exact same subset of packages, no matter what your transitive dependencies are doing. You can use pip-compile from…

I have been using pip since 2014 and did not know about the constraints. This solves my issue with sub-dependency version pinning!

Re: How Python virtual environments work

#274

Post by aflag »

crabbone wrote: post #131
aflag wrote: post #73

Earlier quoted context omitted.

It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppo…

I hate PyPI probably even more than you do, but venv doesn't do that. All it does is write a handful of files and make a bunch of symlinks. It doesn't deal with installation of packages.

Ok. Fair. Venvs are great, unless you want to install packages on them.

Re: How Python virtual environments work

#275

Post by bombolo »

ElectricalUnion wrote: post #264
bombolo wrote: post #258

Earlier quoted context omitted.

You should really start using linux distributions. These problems are all solved and have been solved for a long time.

What happens when your distribution only have old versions, or worse, no versions of the libraries you need? You hoop distribution? You layer another distribution like Nix or Anaconda over your base distribution? You give up and bundle another entire distribution in a container image?

You make a package for the thing you need.

Re: How Python virtual environments work

#276

Post by atoav »

_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I agree with the statement that venvs are usable and fine. However, they do not come without their pitfalls in the greater picture of development and deployment of python software.

It very often not as simple as going to your target system, cloning the repo and running a single line command that gives you the running software. This is what e.g. Rust's cargo would do.

The problem with python venvs is that when problems occur, they require a lot of deep knowledge very fast and that deep knowledge will not be available to the typical beginner. Even I as a decade long python dev will occasionally encounter stuff that takes longer to resolve than needed.

Re: How Python virtual environments work

#277

Post by emptysongglass »

_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

But why bother? Just use PDM in PEP-582 mode [1] which handles packages the same way as project-scoped Node packages. Virtual environments are just a stop-gap that persisted for long enough for a whole ecosystem of tooling to support for them. It doesn't make them less bad, just less frustrating to deal with.

[1] https://pdm.fming.dev/latest/usage/pep582/

Re: How Python virtual environments work

#278

Post by crabbone »

groestl wrote: post #271
crabbone wrote: post #261

Earlier quoted context omitted.

> It solves it by sidestepping the issue Lol. It doesn't. How are you going to create Docker images? Run more Dockers inside Docker? And then Dockers all the way down? You still need to install the packages... just while creating an image. Makes absolutely no difference whether you do it inside or outside, you still need something that can install and coordinate packages. Docker is simply irrelevant to this problem.

> How are you going to create Docker images? Once. On a build server. Or a dev's workstation who had the time to fiddle with all the breakage that is python envs. This absolutely does sidestep the issue of distributing Python packages to your users. We're talking about different problems here. You are talking about library dependencies for devs (and I understand that I'm a bit off topic for this thread)

> Once. On a build server.

I didn't ask where, I asked how. How will you know what packages do you need to install? Are you going to go on pypi.org, download them one by one, unzip, look for requirements, then go to pypi.org again, download more, and so on, until you have enough packages and have figured out all the requirements? If so, I have very bad news for you: your single installation might take weeks, possibly months, or might in fact be so slow it will never end as new packages will be published faster than you can identify the version you need.

> This absolutely does sidestep the issue of distributing Python packages to your users.

No it doesn't... OMG. You cannot be serious? Do you really search and solve dependency problems w/o using any dedicated software? Like in your head? Or with pen and paper? For any non-trivial project the number of combinations you have to try before you find the one that works will be astronomical...

> We're talking about different problems here.

Nah. You just don't understand the problem. It's similar, but not the same.

Re: How Python virtual environments work

#279

Post by mixmastamyk »

ElectricalUnion wrote: post #267
mixmastamyk wrote: post #181

Earlier quoted context omitted.

System administration skills are necessary to be productive developer. There is nothing re: python that can't be fixed with a few shell commands.

If a rogue package rm's your root directory as root, you need a bit more that a few shell commands to fix it.

Can't happen unless you install as root. You're NOT DOING THAT are you?

Also LiveCDs are a thing for about twenty years. Recovery has never been easier, even after hardware failure.

Re: How Python virtual environments work

#280

Post by groestl »

crabbone wrote: post #278
groestl wrote: post #271

Earlier quoted context omitted.

> How are you going to create Docker images? Once. On a build server. Or a dev's workstation who had the time to fiddle with all the breakage that is python envs. This absolutely does sidestep the issue of distributing Python packages to your users. We're talking about different problems here. You are talking about library dependencies for devs (and I understand that I'm a bit off topic for this thread)

> Once. On a build server. I didn't ask where, I asked how . How will you know what packages do you need to install? Are you going to go on pypi.org, download them one by one, unzip, look for requirements, then go to pypi.org again, download more, and so on, until you have enough packages and have figured out all the requirements? If so, I have very bad news for you: your single installation might take weeks, possibl…

You have a very abrasive style of dialog, combined with an unwillingness to consider alternative POVs in your discussion, bordering on trolling. I like discussion, but I don't enjoy this. With that, I will disengage now.
Post reply on HN