Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overview of Python dependency management tools

modelpredict.com

Post reply on HN
Print view View original

171–180 of 184 posts

Re: Overview of Python dependency management tools

#171

Post by darkarmani »

throwaway894345 wrote: post #30
drcongo wrote: post #15

Every attempt to solve this problem in Python seems to eventually end up in a pretty terrible place. Pipenv got off to a great start but got slower and slower to the point that it was more painful to use than not. Poetry (which is still my preferred option) started off with something seemingly beautifully thought through, and very fast too. But after only a few version updates, it seems to be hitting the same problem…

I get that resolving dependencies is a SAT problem and inherently intensive; however, I don't understand why it's so much slower in Python. Is it just that all of these resolvers are implemented in Python (and Python is really that much slower than other languages?), or does Python require you to download an entire package just to determine its dependencies? In the latter case, that seems pretty dumb, right? Like as…

That's why conda stores the metadata in a repodata file. Solving dependencies happens first; then binary packages are downloaded.

Re: Overview of Python dependency management tools

#172

Post by darkarmani »

franciscop wrote: post #60
d0mine wrote: post #51

Earlier quoted context omitted.

pip install works even on iPhone (for pure Python packages in Pythonista for iOS)

Do you mean pip3 install ? Also tried that, didn't work. Had to learn about Python versions, pip vs pip3 versions, pipenv, conda, how an old python package doesn't work with a modern Python package, etc. All I was trying is to combine tensorflow lite with opencv IIRC. Just look at the installation instructions: - https://www.tensorflow.org/install/pip - https://www.tensorflow.org/lite/guide/python - https://docs.open…

I'd love to see how npm would deal with tensorflow. It's a tough package to compile from scratch.

Re: Overview of Python dependency management tools

#173

Post by WD-42 »

darkarmani wrote: post #170
WD-42 wrote: post #134

Earlier quoted context omitted.

The only time I run into problems is when someone else is trying to use Conda. Then it can be hell trying to get their code running in standard pip/venv or vice versa. I'm sure Anacona filled a niche at some point, but we have wheels now, can we all just agree to stop using Conda? What value does it actually bring now that makes it worth screwing up the standard distribution tools?

Isn't a conda environment just python installed into an isolated directory where someone can run pip? One can just run pip and pretend it isn't a conda environment.

It's way more than that. Firstly, most Anaconda installations come shipped with libraries like Matplot, numpy, etc. So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py.

Then there's the issue of Anaconda using it's own package repos, so even if you do manage to figure out what packages an Anaconda developed piece of software needs, you're getting a subtlety or maybe not so subtlety different version of it using standard pip, which creates the worst kind of hard to trace bugs.

Lastly, certain installations of Anaconda overwrite the system python version with it's own (so you can just use numpy or whatever anywhere) causing a huge headache with other system software and making using the standard distribution tools even harder.

I get that it's convenient for scientists that just want to write scripts and have them work, but if you're creating any kind of collaborative software, especially if you'll be working with SW engineers down the line, avoid Conda at all costs.

Re: Overview of Python dependency management tools

#174

Post by kissgyorgy »

FridgeSeal wrote: post #165
kissgyorgy wrote: post #119

> Pipenv or poetry? If you used pipenv for a complex project with huge dependency tree, or used it for a long time, you definitely run into a blocker issue with it. That is the worst package manager of all, and probably the reason why Python has such a bad reputation in this area. It's because it's fundamentals are terrible. Just go with Poetry. It's very stable, easy to use, has a superior dependency resolver and wa…

Am I misunderstanding Poetry? Because it seems to me more suited as something for packaging your python code up ready to be pushed to Pypi? As in, starting up a project creates an init.py, and a python file referencing distutils: neither of which I need or want to do if I'm writing an app to go into a docker container.

The first use case is actually handling project dependencies. If I remember correctly, it couldn't build packages at the first time, so the "build" subcommand was introduced only later. It's the same type of package manager (with lock file) as other languages already had like Cargo, Bundler or NPM.

Re: Overview of Python dependency management tools

#175

Post by darkarmani »

WD-42 wrote: post #173
darkarmani wrote: post #170

Earlier quoted context omitted.

Isn't a conda environment just python installed into an isolated directory where someone can run pip? One can just run pip and pretend it isn't a conda environment.

It's way more than that. Firstly, most Anaconda installations come shipped with libraries like Matplot, numpy, etc. So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py. Then there's the issue of Anaconda using it's own package repos, so even if you do manage to figure out what packages an Anaconda developed piece of so…

> So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py.

How is that any different than using python.org python? You'd still be unaware of what versions to use.

> you're getting a subtlety or maybe not so subtlety different version of it using standard pip, which creates the worst kind of hard to trace bugs.

That's way more of a problem with pip. You have no idea what versions a pip package is pulling in until install and then what binary actually gets installed depends on your compilers.

> certain installations of Anaconda overwrite the system python version with it's own (so you can just use numpy or whatever anywhere) causing a huge headache with other system software and making using the standard distribution tools even harder.

That's impossible unless one is actually copying binaries manually overtop of system binaries. You'd have to be root or use sudo to overwrite the system python manually. The whole point of isolation is to keep system python isolated and stable for system stability. That can happen if someone installs python from python.org and copies it into place.

> but if you're creating any kind of collaborative software, especially if you'll be working with SW engineers down the line, avoid Conda at all costs.

If you are working with SW engineers, you better know what versions you are pulling in, because you are going to be in serious pain using pip and trying to understand the provenance of your packages. Conda is way more powerful here for serious engineers to specify exact versions and reproducible and exact builds.

Re: Overview of Python dependency management tools

#176

Post by mariocesar »

autograd wrote: post #113
mariocesar wrote: post #63

Earlier quoted context omitted.

Yes !! I just create a Makefile target and pip-tools is all I need. I create a requirements.in and that is all. So far never feel that has to be more complicated than that. And when I want to upgrade a package I update the requirements.in if I need to and run `make -B` for this: default: requirements-develop.txt pip install -r requirements-develop.txt requirements.txt: pip-compile -v requirements.in requirements-deve…

Does it pin versions of 2nd degree dependencies too? Like pip freeze would do? Also, when you remove a package, does it know to clear packages that were its deps and are not needed anymore?

Yes, and yes

Re: Overview of Python dependency management tools

#177

Post by dalke »

oblio wrote: post #154
gen220 wrote: post #149

Earlier quoted context omitted.

You're right of course. Still, it's worth keeping in mind that Rust was born 20 years after Python was. Python was being written before Mosaic, Netscape, and Yahoo! were around. I think it can be forgiven for failing to conceive of a perfect package management system in 1990s. There were bigger fish to fry back then, so to speak. Over the decades (!) there have been many, well-documented attempts at coming up with a…

This comment is rewriting Python history quite a bit. First of all, Python was created around 1989 yet Python 1.0 was released in 1994. Secondly, Python was a pretty obscure language until Python 2.0 (and even long after that...), released in 2000. So realistically, Python had "only" about 15 years of historical baggage :-) Also, cargo can be ignored because it's "new", but there was a lot of prior art in the area of…

Python's first posting to Usenet (v 0.9) was in 1991. The 1.0 Misc/ACKS from 1994 includes 50 or so external contributions to that point, showing that "1.0" is a somewhat artificial point.

Rust's 1.0 was 2015, which is indeed "20 years after Python was" at 1.0, so how is gen220's comment a rewrite?

I started using Python around 1.3, and advocating for its broader use (instead of Perl) by 1997. In 1998 I had a job using Python full-time. It was made easier because tools like SWIG already supported Python. Here's a talk I gave in 1999 - https://www.daylight.com/meetings/mug99/Dalke/index.html - and a writeup I did for Dr. Dobb's - https://www.drdobbs.com/cpp/making-c-extensions-more-pythoni... .

In 2000 I helped a company with the minor work to port their 1.5 code base to 2.x.

So I certainly didn't see it as obscure in the 1.x days.

But sure, I'm part of that environment so have a different view on things. If I use your definition, I'll argue that Rust is still "a pretty obscure language".

Re: Overview of Python dependency management tools

#178

Post by oblio »

dalke wrote: post #177
oblio wrote: post #154

Earlier quoted context omitted.

This comment is rewriting Python history quite a bit. First of all, Python was created around 1989 yet Python 1.0 was released in 1994. Secondly, Python was a pretty obscure language until Python 2.0 (and even long after that...), released in 2000. So realistically, Python had "only" about 15 years of historical baggage :-) Also, cargo can be ignored because it's "new", but there was a lot of prior art in the area of…

Python's first posting to Usenet (v 0.9) was in 1991. The 1.0 Misc/ACKS from 1994 includes 50 or so external contributions to that point, showing that "1.0" is a somewhat artificial point. Rust's 1.0 was 2015, which is indeed "20 years after Python was" at 1.0, so how is gen220's comment a rewrite? I started using Python around 1.3, and advocating for its broader use (instead of Perl) by 1997. In 1998 I had a job usi…

Rust is a pretty obscure language, when compared to Java, C#, C/C++, Javascript, Python, etc.

Rust is a lot better known because of internet fame, which didn't really exist to this magnitude until well after the dotcom crash.

My point is, Python's popularity took off in recent years, basically the last 10 years: https://insights.stackoverflow.com/trends?tags=python%2Cjava, primarily due to data science, machine learning, science in general.

Around until 2005 at least, it was known as a friendly scripting language with a few web frameworks which were not that popular (Django was first released in 2005), as the language that was starting to be adopted by distributions for scripting tools (the first Ubuntu version was launched in 2004 and was one of the first distros to use it extensively). It wasn't really present for development work in most cases, DevOps was the domain of bash/Perl (for older stuff) or Ruby (for newer stuff).

People tend to forget how obscure Python was before 2000, compared to the mainstream language it is today. And I say that as someone who likes Python ;-)

Re: Overview of Python dependency management tools

#179

Post by pjs_ »

anxq11 wrote: post #147
emmelaich wrote: post #99

Earlier quoted context omitted.

Can you elaborate?

Perhaps I can guess some causes of the irritation. Let me start by saying that on Windows conda is probably an improvement. On Linux, however, I do not see much benefit, unless you frequently install large binary C library based packages. To me it feels cleaner to compile these packages from source. You are sure to have no glibc mismatches etc. Conda, despite its advertising, does have library issues. C Libraries are…

This is a perfect summary which exactly captures my frustrations.

I should have said that on Windows Conda is a much more understandable choice.

Re: Overview of Python dependency management tools

#180

Post by gen220 »

oblio wrote: post #178
dalke wrote: post #177

Earlier quoted context omitted.

Python's first posting to Usenet (v 0.9) was in 1991. The 1.0 Misc/ACKS from 1994 includes 50 or so external contributions to that point, showing that "1.0" is a somewhat artificial point. Rust's 1.0 was 2015, which is indeed "20 years after Python was" at 1.0, so how is gen220's comment a rewrite? I started using Python around 1.3, and advocating for its broader use (instead of Perl) by 1997. In 1998 I had a job usi…

Rust is a pretty obscure language, when compared to Java, C#, C/C++, Javascript, Python, etc. Rust is a lot better known because of internet fame, which didn't really exist to this magnitude until well after the dotcom crash. My point is, Python's popularity took off in recent years, basically the last 10 years: https://insights.stackoverflow.com/trends?tags=python%2Cjava , primarily due to data science, machine lear…

Chiming back in to say that, while everything you're saying is correct (i.e. Python was not a ubiquitous language until "relatively" recently), it doesn't change the point: the best packaging solutions, done right, need to be done early in a language's history.

To illustrate the point with an example, you could invent cargo for python yesterday or in 2005, but it wouldn't have solved the problem, because you would still have decades-worth of third-party libraries that wouldn't comply to py-cargo's packaging requirements.

In contexts like these, it's the package manager with the fewest hard-asks (i.e. pip, or npm for node) that wins.

Go, for example, endured major controversies over migrating away from GOPATH-managed-with-third-party-dep-managers to go modules. Even though `go mod` would have been the best solution to start with from scratch, inertia and breaking changes are a real thing.

Post reply on HN