Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New Pip resolver takes a long time to complete

github.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: New Pip resolver takes a long time to complete

#31

Post by ryukafalz »

cbaines wrote: post #11

I used to think package managers and dependency resolvers were inseperable, I even did a little bit of work on the APT resolver. Since using GNU Guix though, I'm so glad it doesn't have a dependency resolver as part of building or installing packages! It's so much better for it, no slow or unpredictable resolving, you know what it's going to do. I think this is one reason why I've never used pip for managing Python s…

As a Guix user/occasional packager who has run into broken packages a few times (including Python ones)... I do wish it had a dependency resolver that could at least be run outside the build/install process, e.g. when updating libraries.

If I'm updating a library that has a bunch of dependent packages, it's hard to know whether or not something will break downstream. Sometimes this is unavoidable and you really do need to just test everything thoroughly, but sometimes the dependent packages have more knowledge of what library versions they need... and Guix doesn't seem to be aware of this.

Re: New Pip resolver takes a long time to complete

#32

Post by yjftsjthsd-h »

cbaines wrote: post #22
yjftsjthsd-h wrote: post #17

Earlier quoted context omitted.

Wait, how does guix avoid a dependency resolver? It still has dependencies for packages... are they just hardcoding exact dependencies (including versions) and relying on the "can install multiple versions of a package" property to make that work? That seems inefficient, though I guess maybe they need that for the "this hash means this exact binary" outcome?

Guix has packages, and packages have inputs (like dependencies), and you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). Guix package definitions are truely code though, so if you want to generate packages on the fly by using a dependency resolver, you can totally write some code to make that happen. With respect to inefficiency, what do you mean? It's qui…

> With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies.

It's going to burn disk space like crazy, isn't it? If I install packages foo-1.0 and bar-1.0 and foo-1.0 uses glibc-2.31 and bar-1.0 uses glibc-2.30 then I now have 2 versions of glibc... but that scales to every package I install and every library every one of them uses. Of course, this can be fixed... by automatically building new versions of every package any time any of their dependencies changes, in which case we're probably not wasting much disk because we traded and are now burning CPU time like there's no tomorrow (and network, and disk I/O, and memory, and anything else used in package builds). Basically, this sounds like reinventing static binaries, with all the downsides thereof.

Re: New Pip resolver takes a long time to complete

#33

Post by martin_renou »

_3sno wrote: post #24
rdlfo wrote: post #3

Perhaps something like mamba [1] could speed up the resolver. That is if the resolver is actually achieving anything at all. [1] https://github.com/mamba-org/mamba

My understanding is mamba, like conda, just call pip. So it likely wouldn't make a difference. The pip section in a env file is just a list of arguments passed through to the pip install command. Prior to pip 20.3 we had to add `--use-feature=2020-resolver` to get an install that resolved for our teams that used mamba.

Mamba and conda do not call pip.

You can install a pip package inside a conda environment. But when running `mamba install` or `conda install`, pip is not involved at all.

Re: New Pip resolver takes a long time to complete

#34

Post by martin_renou »

_3sno wrote: post #29
Iwan-Zotow wrote: post #27

Earlier quoted context omitted.

No, conda is not calling pip

You're wrong, it does. Conda installs conda packages and conda uses pip to install pip packages. However a pip package can be converted to a conda package, and then in that case the dependency will be installed by conda and not pip.

You're wrong, it does not.

You can install a pip package in a conda env. But this is actually not recommended.

When using `mamba install` or `conda install`, pip is not involved at all.

Re: New Pip resolver takes a long time to complete

#35

Post by uranusjr »

martin_renou wrote: post #34
_3sno wrote: post #29

Earlier quoted context omitted.

You're wrong, it does. Conda installs conda packages and conda uses pip to install pip packages. However a pip package can be converted to a conda package, and then in that case the dependency will be installed by conda and not pip.

You're wrong, it does not. You can install a pip package in a conda env. But this is actually not recommended. When using `mamba install` or `conda install`, pip is not involved at all.

You’re correct, Conda does not call pip when it installs packages. But pip does have a subtle involvement here: Many Python packages available to Conda are packaged based on installations made by pip, and (IMO lazily and incorrectly) inherits a lot of pip characteristics. This makes the packages “appear” to be installed by pip, and sometimes induce interpolation inconsistencies.

Re: New Pip resolver takes a long time to complete

#36

Post by orf »

sjburt wrote: post #9

They need to store the version requirements metadata outside the packages. Having to download the entire wheel just to see whether it is compatible is ridiculous. This could all be computed by downloading an index file, then performing the resolution. They made a half-assed attempt when pypa/pip added the "data-requires-python" tag, but that covers python interpreter version only, it needed to have been done for all…

Pypi does have a json API that returns package dependencies, if the uploaded file is compatible.

See the “requires’ key here: https://pypi.org/pypi/Django/json

Re: New Pip resolver takes a long time to complete

#37

Post by cbaines »

jrochkind1 wrote: post #26
cbaines wrote: post #22

Earlier quoted context omitted.

Guix has packages, and packages have inputs (like dependencies), and you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). Guix package definitions are truely code though, so if you want to generate packages on the fly by using a dependency resolver, you can totally write some code to make that happen. With respect to inefficiency, what do you mean? It's qui…

If they are specifying exact hard-coded dependency versions... if a X -> Y -> A, and B -> A, and M -> N -> A... and a security patch is released for A, then X, Y, B, M and N all need new releases to specify new exact dependencies, in order to use the new A'? It's disastrous for security patches, only highly inconvenient for things like performance improvement releases. But this is why we have dependency resolving, ri…

No, you just update the package definition for A and release the updated package definitions, people will then be using the updated A, whether directly or through other packages (X, Y, B, M, N, ...).

There is an issue here of rebuilding all those dependent packages with the updated A, especially if it's something like glibc. Guix includes a mechanism called grafts that allows for package replacements, which allows avoiding this, and this is often used for releasing security fixes.

Re: New Pip resolver takes a long time to complete

#38

Post by cbaines »

yjftsjthsd-h wrote: post #32
cbaines wrote: post #22

Earlier quoted context omitted.

Guix has packages, and packages have inputs (like dependencies), and you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). Guix package definitions are truely code though, so if you want to generate packages on the fly by using a dependency resolver, you can totally write some code to make that happen. With respect to inefficiency, what do you mean? It's qui…

> With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies. It's going to burn disk space like crazy, isn't it? If I install packages foo-1.0 and bar-1.0 and foo-1.0 uses glibc-2.31 and bar-1.0 uses glibc-2.30 then I now have 2 versions of glibc... but that scales to every package I install and every library every one of them…

Normally you'd just be using one glibc version. Packages that you installed a while ago may be using a different one, but it's not like every package has massively divergent dependencies.

Because of the immutable store, you can do file level deduplication, so if you have multiple versions of the same packages, you can deduplicate the identical files.

I think the worries about disk usage are relevant, but only on systems with small amounts of storage. These are still relevant though, and it's an important area to improve on. As for burning CPU time, I don't think there's a perfect solution to avoid this, but I think Guix is pretty good. Guix provides substitutes, so you don't have to build things locally on every machine (I'm looking at you Rubygems, pip and Python stuff is pretty bad also).

Re: New Pip resolver takes a long time to complete

#39

Post by FRidh »

ryukafalz wrote: post #31
cbaines wrote: post #11

I used to think package managers and dependency resolvers were inseperable, I even did a little bit of work on the APT resolver. Since using GNU Guix though, I'm so glad it doesn't have a dependency resolver as part of building or installing packages! It's so much better for it, no slow or unpredictable resolving, you know what it's going to do. I think this is one reason why I've never used pip for managing Python s…

As a Guix user/occasional packager who has run into broken packages a few times (including Python ones)... I do wish it had a dependency resolver that could at least be run outside the build/install process, e.g. when updating libraries. If I'm updating a library that has a bunch of dependent packages, it's hard to know whether or not something will break downstream. Sometimes this is unavoidable and you really do ne…

Exactly. Python maintainer of Nixpkgs here.

What we need is to be able to use a resolver such as included by pip or poetry, to build up our package set. In Nixpkgs this is nowadays unfortunately done manually, and I suppose the same goes for Guix. In Nixpkgs the reason is simple: too eager pinning makes it impossible to resolve a package set that works with the entire set.

Now that pip has a resolver what is needed is a way to use constraints not to set only lower and upper bounds, but to enforce a version when resolving. That makes it usable for downstream integrators to construct their primary package set. One could then even make the next step and construct "stable" sets that extend the primary set.

Re: New Pip resolver takes a long time to complete

#40

Post by rdlfo »

martin_renou wrote: post #33
_3sno wrote: post #24

Earlier quoted context omitted.

My understanding is mamba, like conda, just call pip. So it likely wouldn't make a difference. The pip section in a env file is just a list of arguments passed through to the pip install command. Prior to pip 20.3 we had to add `--use-feature=2020-resolver` to get an install that resolved for our teams that used mamba.

Mamba and conda do not call pip. You can install a pip package inside a conda environment. But when running `mamba install` or `conda install`, pip is not involved at all.

I got a bit confused by the statements in the thread. If the issue is downloading wheels to check dependencies, mamba should be a better alternative, as not only it is downloading packages in parallel but it uses a different dependency solver.
Post reply on HN