New Pip resolver takes a long time to complete • Hacker News Board

Re: New Pip resolver takes a long time to complete

#41

Post by jrochkind1 » Fri, Dec 04, 2020, 3:32 PM UTC

cbaines wrote: post #37

jrochkind1 wrote: post #26

Earlier quoted context omitted.

If they are specifying exact hard-coded dependency versions... if a X -> Y -> A, and B -> A, and M -> N -> A... and a security patch is released for A, then X, Y, B, M and N all need new releases to specify new exact dependencies, in order to use the new A'? It's disastrous for security patches, only highly inconvenient for things like performance improvement releases. But this is why we have dependency resolving, ri…

No, you just update the package definition for A and release the updated package definitions, people will then be using the updated A, whether directly or through other packages (X, Y, B, M, N, ...). There is an issue here of rebuilding all those dependent packages with the updated A, especially if it's something like glibc. Guix includes a mechanism called grafts that allows for package replacements, which allows av…

I think I must be misunderstanding. When you said:

> you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded).

I thought that meant that package X might specify a dependency on A version 2.4.2 exactly. So if you want it to use A 2.4.3 instead, a new release of package X would have to be created, specifying 2.4.3.

But I think this is not in fact what you mean? In which case I don't yet understand the system you are describing and what you mean by not doing dependency resolution.

Re: New Pip resolver takes a long time to complete

#42

Post by microtonal » Fri, Dec 04, 2020, 4:09 PM UTC

jrochkind1 wrote: post #41

cbaines wrote: post #37

Earlier quoted context omitted.

No, you just update the package definition for A and release the updated package definitions, people will then be using the updated A, whether directly or through other packages (X, Y, B, M, N, ...). There is an issue here of rebuilding all those dependent packages with the updated A, especially if it's something like glibc. Guix includes a mechanism called grafts that allows for package replacements, which allows av…

I think I must be misunderstanding. When you said: > you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). I thought that meant that package X might specify a dependency on A version 2.4.2 exactly. So if you want it to use A 2.4.3 instead, a new release of package X would have to be created, specifying 2.4.3. But I think this is not in fact what you mean? In…

I thought that meant that package X might specify a dependency on A

In Nix and Guix, you would indeed not specify that X needs package A version 2.4.2. You could see it as X using the 'build recipe' of A as its dependency. So, if you bump the version of A to 2.4.3, then all packages that use A as a dependency will be rebuilt (or substituted from the a binary cache if the Guix/NixOS build infrastructure has already built the updated packages).

Re: New Pip resolver takes a long time to complete

#43

Post by nawgz » Fri, Dec 04, 2020, 5:42 PM UTC

uranusjr wrote: post #28

nawgz wrote: post #16

Earlier quoted context omitted.

>IMO Python has one of the more mature packaging ecosystems out there Ouch, what languages are you using where this is true? For example, JS / NPM / Yarn are absolutely blowing pip out of the water. I guess I can imagine Java or C++ users having your perspective though

Every language community has its own priorities, and the package manager grows different traits to meet community needs. Those traits would come at a price, and different comunnities (and their package managers) would make different tradeoffs due to their different priorities. Python has a very involved history dealing with platform-native stuff, and provide a lot of convinience aroud specifying, providing, and obtai…

>The only package manager you think is good is from the ecosystem

Or maybe, the only package manager you think is good is the one that has features you value and doesn't fail in ways you wouldn't expect?

For example, I have been trying to work with numpy and pandas a bit - two of the biggest Python libraries - and use them on my MacBook (itself a popular item). These installs fail, in the middle of an ungrokable stack of install logs. I have to shuffle thru useless messages to eventually track down the source, and then try to find a new compatible version. So sure, maybe there's native code in there, but I think claiming "useful summaries are not the package manager's job" is silly

But I also have had other terrible experiences: packaging is a bit overwrought; the terrible import/export/module system in Python means your dependency names have nothing to do with where you import from; SSL Certificate errors crop up at random on domains with valid certificate.

I am sure it's good enough to use, but I think it's a bit pie-in-the-sky to claim all package managers are good and you just have to be in the community. Shitty software exists

Re: New Pip resolver takes a long time to complete

#44

Post by jrochkind1 » Fri, Dec 04, 2020, 5:43 PM UTC

microtonal wrote: post #42

jrochkind1 wrote: post #41

Earlier quoted context omitted.

I think I must be misunderstanding. When you said: > you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). I thought that meant that package X might specify a dependency on A version 2.4.2 exactly. So if you want it to use A 2.4.3 instead, a new release of package X would have to be created, specifying 2.4.3. But I think this is not in fact what you mean? In…

I thought that meant that package X might specify a dependency on A In Nix and Guix, you would indeed not specify that X needs package A version 2.4.2. You could see it as X using the 'build recipe' of A as its dependency. So, if you bump the version of A to 2.4.3, then all packages that use A as a dependency will be rebuilt (or substituted from the a binary cache if the Guix/NixOS build infrastructure has already bu…

Huh, in that case I have the converse question -- when specifying dependencies for X, can you not say A 1.x but not 2.x, because 2.x has or is expected to have backwards breaking changes? Otherwise, when A releases 2.0 with some backwards incompatibilities, and all packages that use A as a dependency are rebuilt, don't they break?

These issues (allow updates within limits) are what I understand as the point of dependency resolution, I'm trying to understand how you do without it.

Re: New Pip resolver takes a long time to complete

#45

Post by microtonal » Sat, Dec 05, 2020, 7:31 AM UTC

jrochkind1 wrote: post #44

microtonal wrote: post #42

Earlier quoted context omitted.

I thought that meant that package X might specify a dependency on A In Nix and Guix, you would indeed not specify that X needs package A version 2.4.2. You could see it as X using the 'build recipe' of A as its dependency. So, if you bump the version of A to 2.4.3, then all packages that use A as a dependency will be rebuilt (or substituted from the a binary cache if the Guix/NixOS build infrastructure has already bu…

Huh, in that case I have the converse question -- when specifying dependencies for X, can you not say A 1.x but not 2.x, because 2.x has or is expected to have backwards breaking changes? Otherwise, when A releases 2.0 with some backwards incompatibilities, and all packages that use A as a dependency are rebuilt, don't they break? These issues (allow updates within limits) are what I understand as the point of depend…

These issues (allow updates within limits) are what I understand as the point of dependency resolution, I'm trying to understand how you do without it.

In such a case e.g. nixpkgs makes two attributes: a_1 and a_2 (and alias a to a_2). Packages that still require A 1.x will have a_1 as one of their dependencies, the rest a.

This is avoided as much as possible, but is sometimes necessary. Common examples are Gtk 2 applications or C/C++ applications that can only be built against a Python 2 interpreter.