Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

261–270 of 293 posts

Re: How Python virtual environments work

#261

Post by crabbone »

groestl wrote: post #232
crabbone wrote: post #162

Earlier quoted context omitted.

Docker is not the answer to any packaging problems because it's not a packaging tool. I have no idea how people don't understand this... oh wait! I'm talking to Python programmers! But... I'm not going to hold you in the dark: the reason and the major drive to have a packaging system is that you can define dependencies between packages s.t. users installing package can coordinate and install the stuff they need. Dock…

Your post reminds me of the saying with the faster horses. Docker doesn't solve Python's packaging problems by being a better packaging system. It solves it by sidestepping the issue of users installing packages themselves, and shipping the disk of the one machine where it once worked. It's a bad linker. (not a fan of the condescending tone in your post, btw)

> It solves it by sidestepping the issue

Lol. It doesn't. How are you going to create Docker images? Run more Dockers inside Docker? And then Dockers all the way down? You still need to install the packages... just while creating an image. Makes absolutely no difference whether you do it inside or outside, you still need something that can install and coordinate packages. Docker is simply irrelevant to this problem.

Re: How Python virtual environments work

#262

Post by crabbone »

woodruffw wrote: post #173
crabbone wrote: post #134

Earlier quoted context omitted.

> once or twice have I ever encountered a showstopper dependency resolution issue. Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways! Until quite recently, pip didn't give a rat's ass if the dependencies it installe…

I understand that you're being hyperbolic for rhetorical purposes, but I think you're overselling the problem with source distributions: most language package ecosystems boil down to the same "baseline" package representation as sdists do, and have the same basic "build it if no binary matches" approach. Many don't even provide built distributions; Rust and Go come to mind. Python's problem isn't with source distribu…

> Python's problem isn't with source distributions as such, but with really bad metadata control

One doesn't preclude the other. I'm not against having a mechanism for automating source installs (like this is done in, eg. RHEL-based distros), but it's insanity if you allow this to happen by default. You may not remember Bumblebee deleting /usr while running some innocuously-looking code during install, but things happen... really bad things...

Things don't need to happen all the time in order for them to be scary. It's enough to have possible catastrophic consequences, even if the event itself is rare.

> Better metadata makes source-based language package management work just fine in every other language's ecosystem

I haven't seen a single one, and I used dozens at this point. This is never a good idea. It's OK to do source installs for development, it's never a good idea to do source installs for deployment. It "works" in other places because of how it's presented (i.e. nobody expects this to be the method of software delivery to the end user). Like, eg. in Cargo (Rust): you, as a developer, download sources and build programs from all the sources on your computer, but your user gets a binary blob they put on the system path and run. It would be insanity and a security nightmare if users were supposed to compile program code before they could run it. The select few who can audit what's being downloaded and how it's been compiled would probably manage, the rest would become victims of all sorts of scams or just random failures propagating beyond their builds into their systems.

> much of the effort in Python packaging over the last ~8 years has been slowly turning Python in that direction.

I'm sorry, but PyPA is managed by clueless people. Whatever they do there only breeds more insanity over time. They neither have a general direction where they want to take the packaging system, nor do they understand the fine details of it. They are also bombarded by insane requirements for useless and harmful features, which they often quick to implement... It's a circus what's going on there. I've lost hope years ago, and now I've become an accelerationist. I just like to see it burn and people run around screaming while their backs are on fire. I get paid to fix this mess. So, PyPAs incompetence is my job security.

Re: How Python virtual environments work

#263

Post by crabbone »

akprasad wrote: post #167
crabbone wrote: post #129

Earlier quoted context omitted.

Lol. You put "simple" and "requirements.txt" unironically next to each other... I mean, I think you genuinely believe that what you suggest is simple... so, I won't pretend to not understand how you might think that. I'll explain: There's simplicity in performing and simplicity of understanding the process. It's simple to make more humans, it's very hard to understand how humans work. When you think about using pip w…

What are specific problems you have with pip + requirements.txt, and why do you believe storing links to wheels is more reliable? Your comment makes your conclusion clear, but I don't follow your argument.

Pip is a huge and convoluted program with tons of bugs. It does a lot more than just download Python packages and unpack them into their destination. Obviously, if you want something simple, then HTTP client, which constitutes only a tiny fraction of pip would be a simpler solution, wouldn't it?

In practice, pip may not honor your requirements.txt the way you think it would. Even if you require exact versions of packages (which is something you shouldn't do for programs / libraries). This is because pip will install something first, with its dependencies, and then move to the next item, and then this item may or may not match what was already installed.

The reason you don't run into situations like this one often enough to be upset is because a lot of Python projects don't survive for very long. They become broken beyond repair after few years of no maintenance. Where by maintenance I mean constant chasing of the most recent set of dependencies. Once you try to install and older project using pip and requirements.txt, it's going to explode...

Re: How Python virtual environments work

#264

Post by ElectricalUnion »

bombolo wrote: post #258
MrJohz wrote: post #244

Earlier quoted context omitted.

I mean, a project needs regular care and maintenance, however you organise it. If you're never scheduling time to maintain your dependencies, you're going to be in trouble either way. But at least if you lock your dependencies, you know what will actually get installed, and you can find the buggy or insecure versions. We found a bug on a Python project I worked on recently that only seemed to happen on certain machin…

You should really start using linux distributions. These problems are all solved and have been solved for a long time.

What happens when your distribution only have old versions, or worse, no versions of the libraries you need? You hoop distribution? You layer another distribution like Nix or Anaconda over your base distribution? You give up and bundle another entire distribution in a container image?

Re: How Python virtual environments work

#265

Post by ElectricalUnion »

coldtea wrote: post #249
chpatrick wrote: post #172

Earlier quoted context omitted.

It was pretty bad before but now it seems like there are a bunch of competing solutions each with their own quirks and problems. It feels like the JavaScript ecosystem.

The Javascript ecosystem for other things, like frameworks, sure. But when it comes to packages and "virtual envs" the Javascript ecosystem is leaps and bounds better.

The "Javascript ecosystem" on my personal experience seems to prefeer installing everything in the global environment "for ease of use convenience" and then they wonder how did a random deprecated and vulnerable dependency get inside their sometimes flattened, sometimes nested, non-deterministic dependency chain (I wish the deterministic nested pnpm was the standard...) and (pretend) they did not notice.

That being said, the Javascript ecosystem has standarized tooling to handle that (npx) that Python doesn't (I wish pipx was part of standard pip), they just pick the convenient footgun approach.

Re: How Python virtual environments work

#266

Post by [deleted] »

kermatt wrote: post #213
anongraddebt wrote: post #151

Earlier quoted context omitted.

Twice bricking my laptop’s ability to do python development because of venv + symlink bs was the catalyst I needed to go all-in on remote dev environments. I don’t drive python daily, but my other projects thank Python for that.

How do you brick a machine with venvs?

[deleted]

Re: How Python virtual environments work

#267

Post by ElectricalUnion »

mixmastamyk wrote: post #181
anongraddebt wrote: post #151

Earlier quoted context omitted.

Twice bricking my laptop’s ability to do python development because of venv + symlink bs was the catalyst I needed to go all-in on remote dev environments. I don’t drive python daily, but my other projects thank Python for that.

System administration skills are necessary to be productive developer. There is nothing re: python that can't be fixed with a few shell commands.

If a rogue package rm's your root directory as root, you need a bit more that a few shell commands to fix it.

Re: How Python virtual environments work

#268

Post by ElectricalUnion »

bombolo wrote: post #238
JohnFen wrote: post #103

Earlier quoted context omitted.

Honestly, virtual environments are one of the reasons why I prefer to avoid Python whenever I can.

why? Also, you don't have to use them.

Well, if develop and ship in a vm/container, you don't have to do it in your system /s

Re: How Python virtual environments work

#269

Post by JohnFen »

bombolo wrote: post #238
JohnFen wrote: post #103

Earlier quoted context omitted.

Honestly, virtual environments are one of the reasons why I prefer to avoid Python whenever I can.

why? Also, you don't have to use them.

It strikes me that virtual environments are a fairly ugly hack to smooth over the fact that Python is not a stable language. It changes a lot, requiring the use of particular runtimes for particular Python code, requiring the installation of multiple runtimes.

That's a pretty serious downside to the language. Virtual environments are needed to help people deal with that downside.

Re: How Python virtual environments work

#270

Post by ElectricalUnion »

Joker_vD wrote: post #78
dangerlibrary wrote: post #15

Earlier quoted context omitted.

I hate python package management - I really do. But I've never actually had a problem with virtual environments, and I think it's because I just use virtualenv directly (rather than conda or whatever else). I have these aliases in my .bashrc, and I can't remember the last time I had a major issue. alias venv='rm -rf ./venv && virtualenv venv && source ./venv/bin/activate' alias vact='source ./venv/bin/activate' alias…

> source ./venv/bin/activate To this day I'm not quite sure why the venv developers decided that sourcing was a good idea; all it does can be effectively replaced with #!/bin/sh export VIRTUAL_ENV="path to venv" export PATH="$VIRTUAL_ENV/bin:$PATH" unset PYTHONHOME exec "$SHELL" Just run this script to get into an "activated" shell. To deactivate, just press Ctrl+D. If you're really fancy, you can replace the last li…

This technique is simple, doesn't reinvent the wheel, and even has it's own name: Bernstein chaining.

http://www.catb.org/~esr/writings/taoup/html/ch06s06.html

Post reply on HN