Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python virtual environments work

snarky.ca

Post reply on HN
Print view View original

121–130 of 293 posts

Re: How Python virtual environments work

#121

Post by 9dev »

rowanseymour wrote: post #98
9dev wrote: post #97

Earlier quoted context omitted.

Oh, yeah? It’s working great? Like figuring out which packages your application actually uses? Or having separate development and production dependencies? Upgrading outdated libraries? Having taken a deep-dive into refactoring a large python app, I can confidently say that package management in python is a pain compared to other interpreted languages.

Virtual environments aren't package management. For example we use Poetry for package management - it supports separate dev and prod dependencies, upgrading etc. It generates a virtual environment.

The distinction feels entirely academic to me. Managing packages means having a sane way to define the dependencies of software projects, so they can be checked into version control and be installed reproducibly later and/or elsewhere.

I don’t know which problem python intended to solve by separating the two, but it doesn’t occur often in contemporary software engineering work.

Having said that, the point you make is valid and Poetry is a good option, but it feels so maddening having to learn about like seven different tools which all do more or less the same but not quite, and everyone and their mother having an opinion on which is the best. Doesn’t help that there’s an arbitrary yet blurry line where package managers end and environment managers begin.

Re: How Python virtual environments work

#122

Post by peterhil »

9dev wrote: post #97
_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

Oh, yeah? It’s working great? Like figuring out which packages your application actually uses? Or having separate development and production dependencies? Upgrading outdated libraries? Having taken a deep-dive into refactoring a large python app, I can confidently say that package management in python is a pain compared to other interpreted languages.

I strongly agree with this, and I have been actively using Python since 2009.

Trying top keep a Pygame/Numpy/Scipy project working has been a real struggle. I started it with Python 2 and ported to Python 3 some years ago. The whole Python 3 transition is a huge mess with every Python 3 point release breaking some things. No other interpreted language’s packaging system is so fucked up.

On a positive note: Lately I've liked using pdm instead of pip, and things seem to work quite a lot better. I evaluated Poetry, Flit and something else also.

I just commented about this on Twitter, when someone asked “Which programming language do you consider beginner's friendly?” https://twitter.com/peterhil/status/1633793218411126789

Re: How Python virtual environments work

#123

Post by justinsaccount »

birdstheword5 wrote: post #24
Max_Limelihood wrote: post #8

Answer: they don’t (Seriously, I’ve gotten so fed up with Python package management that I just use CondaPkg.jl, which uses Julia’s package manager to take care of Python packages. It is just so much cleaner and easier to use than anything in Python.)

It sounds mean to say it, but it's 100% true. I moved away from using python wherever I can. I've had colleagues struggle for days to install well used packages like pandas and numpy in conda.

Things like pandas and numpy are not python packages. Yes, they are packages FOR python, but they are not python.

https://hpc.guix.info/blog/2021/09/whats-in-a-package/ does a good job of explaining why installing packages like that is a complete shitshow.

Re: How Python virtual environments work

#124

Post by schainks »

bobbylarrybobby wrote: post #64
buildbot wrote: post #63

I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.

If it were really that simple, surely all these other solutions wouldn't exist?

If software were that simple, surely all these other languages wouldn't exist?

Re: How Python virtual environments work

#125

Post by peterhil »

wheelerof4te wrote: post #119
pbronez wrote: post #80

Earlier quoted context omitted.

Until you want to use anything with a c extension..

Why would you want to use Python with a C extension? If you need performance, just use native code.

Numpy and Scipy are good reasons. Unfortunately Scipy does not even compile on FreeBSD lately, and I have opened three issues about it against Scipy and Pythran (and the fix was with xsimd).

https://github.com/serge-sans-paille/pythran/issues/2070

Re: How Python virtual environments work

#126

Post by rxhernandez »

whalesalad wrote: post #106
aflag wrote: post #73

Earlier quoted context omitted.

It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppo…

I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager.…

> only once or twice have I ever encountered a showstopper dependency resolution issue

I've encountered them with other languages and they're the sort of thing where one time is more than enough to make me feel like it could get me fired; they're Never (with a capital N) okay imo

Re: How Python virtual environments work

#127

Post by peterhil »

tomalaci wrote: post #49

I would highly recommend Poetry for python package management. It basically wraps around pip and venvs offering a lot of convenience features (managing packages, do dist builds, etc.). It also works pretty nicely with Tox. I would recommend using virualenvs.in-project setting so Poetry generates venv in the project folder and not in some temporary user folder.

I just compared and evaluated Hatch, Flit, Poetry and Pdm and found Pdm to be most robust and slimmest. Hatch was a good second option, and Poetry and Hatch are easy to use, but have too much bloat and magic.

Re: How Python virtual environments work

#128

Post by justinsaccount »

okeuro49 wrote: post #81
adgjlsfhk1 wrote: post #71

Earlier quoted context omitted.

Yeah, one of Julia's best decisions was taking heavy inspiration from Rust for the package manager. Rust was 100% the first language to get dependency management right.

> Rust was 100% the first language to get dependency management right. In my experience, Java, Go, PHP, NodeJS have all got similar package management that works.

See my above comment, there's a reason why all of your examples work:

Java package managers tend to install packages written in java

Go installs packages written in go, and maybe C using cgo

Cargo installs packages written in rust

php package managers install packages written in PHP, extra extensions are rare

etc

People having trouble with python are NOT having trouble with python. They are having trouble because they are trying to use packages that are just python bindings to absolutely massively complex c++ and Fortran libraries.

Often people using python don't even have a C compiler installed (let alone a fortran one for the scientific stuff), so pip blows up the first time they try to install a package that hasn't been pre-built for their system+python version.

Re: How Python virtual environments work

#129

Post by crabbone »

whalesalad wrote: post #104
_coveredInBees wrote: post #67

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools…

I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.

Lol. You put "simple" and "requirements.txt" unironically next to each other...

I mean, I think you genuinely believe that what you suggest is simple... so, I won't pretend to not understand how you might think that. I'll explain:

There's simplicity in performing and simplicity of understanding the process. It's simple to make more humans, it's very hard to understand how humans work. When you think about using pip with requirements.txt you are doing the simple to perform part, but you have no idea what stands behind that.

Unfortunately for you, what stands behind that is ugly and not at all simple. Well, you may say that sometimes it's necessary... but, in this case it's not. It's a product of multiple subsequent failures of people working on this system. Series of mistakes, misunderstandings, bad designs which set in motions processes that in retrospect became impossible to revert.

There aren't good ways to use Python, but even with what we have today, pip + requirements.txt is not anywhere near the best you can do, if you want simplicity. Do you want to know what's actually simple? Here:

Store links to Wheels of your dependencies in a file. You can even call it requirements.txt if you so want. Use curl or equivalent to download those wheels and extract them into what Python calls "platlib" (finding it is left as an exercise for the reader) removing everything in scripts and data catalogues. If you feel adventurous, you can put scripts into the same directory where Python binary is installed, but I wouldn't do that if I were you.

Years of being in infra roles taught me that this is the most reliable way to have nightly builds running quietly and avoiding various "infra failures" due to how poorly Python infra tools behave.

Re: How Python virtual environments work

#130

Post by dima55 »

PaulHoule wrote: post #51
lockhouse wrote: post #45

Earlier quoted context omitted.

Isn’t it considered best practice not to use brew to install Python for exactly this reason? I’ve always seen it recommended to use pyenv or just download directly from Python.org instead.

Python has always struggled with distribution maintainers that just don’t get it. Python barely survived becoming the default scripting language on Red Hat and other Linux distros which was a major obstruction to the Python 3 transition. If the new cohort of pandas and scikit-learn users had not been such a force of nature we”d be talking today about Python the way we do about Perl. Not installing venv is a serious c…

What's the crime in Debian? The "right answer about how to manage packages" is "apt install package". Anything else (including venvs) is a hack
Post reply on HN