Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

121–130 of 152 posts

Re: Freezing Python’s Dependency Hell

#121

The author's justifications for using this home-grown tool over miniconda are weak at best, if not plain incorrect. Conda really is the tool he wants; he just seems not to understand that.

> if not plain incorrect

The justification was that the Anaconda installer is too heavy. The kitchen sink Anaconda installer is not designed for the author's use case. Miniconda is the provided way to bootstrap conda onto a system.

Re: Freezing Python’s Dependency Hell

#122
post #98

Here we go again. The source of the problems in in toy package managers (and I include all language package managers here) is not just the package managers themselves, it's the "version soup" philosophy they present to the user. Not daring to risk displeasing the user, they will take orders akin to "I'd like version 1.2.3 of package a, version 31.4.1q of package b, version 0.271 of package c, version 141 of package d…

There's been quite a bit of discussion about Anaconda and conda in this thread already. Anaconda also takes this distribution approach, and it's targeted specifically at python.

Re: Freezing Python’s Dependency Hell

#123

Earlier quoted context omitted.

That's not true. Python has C libraries, some might need to be built from source, and there's good reason to not allow root access on a lot of systems (and ability to install headers/dev packages, gcc, etc). System package management is hard and coordinating with (ubiqitous, not specific to Python) language package managers magnifies it. Unless you had some other solution in mind that I've missed...

conda basically proves that you can install almost everything one needs in the user's home directory. they have been working more and more on being completely independent from things like system compilers as well.

Conda is a great way to get gcc 7.2 on CentOS 6. Anaconda builds all of its packages targeting CentOS 6 for broad compatibility, but with the latest compilers to ensure we have the latest security features compiled in.

Re: Freezing Python’s Dependency Hell

#124
post #98

Here we go again. The source of the problems in in toy package managers (and I include all language package managers here) is not just the package managers themselves, it's the "version soup" philosophy they present to the user. Not daring to risk displeasing the user, they will take orders akin to "I'd like version 1.2.3 of package a, version 31.4.1q of package b, version 0.271 of package c, version 141 of package d…

There's been quite a bit of discussion about Anaconda and conda in this thread already. Anaconda also takes this distribution approach, and it's targeted specifically at python.

Yet it will never be able to solve the system-library dependency problem in the way that Nix does.

Re: Freezing Python’s Dependency Hell

#125
post #124

Earlier quoted context omitted.

There's been quite a bit of discussion about Anaconda and conda in this thread already. Anaconda also takes this distribution approach, and it's targeted specifically at python.

Yet it will never be able to solve the system-library dependency problem in the way that Nix does.

It solves this already and has done for many years (but this depends on what you mean exactly by "the way that Nix does").

Re: Freezing Python’s Dependency Hell

#127
post #98

Here we go again. The source of the problems in in toy package managers (and I include all language package managers here) is not just the package managers themselves, it's the "version soup" philosophy they present to the user. Not daring to risk displeasing the user, they will take orders akin to "I'd like version 1.2.3 of package a, version 31.4.1q of package b, version 0.271 of package c, version 141 of package d…

Some background: A few months back I was curious about the nix style of packaging so I setup a python project using nix via nixpkgs' pythonPackages. This worked pretty well, but I kept wondering to myself if it was superior to explicitly declaring each version of a package via npm, cargo, bundler, etc.

The way to "freeze" dependencies seemed to involve using a specific git sha of nixpkgs.

From the point of view of a nix newbie, it seems that by relying on nixpkgs to remain relatively stable, you are at the mercy of your package maintainers who might introduce a backwards incompatible change resulting in a build breaking.

One of the alternatives to this was to essentially copy the nix package descriptions from nixpkgs to a projects repo to ensure that packages are explicitly declared. At this point, it felt as though I was maintaining a .lock file by hand.

Do you think nixpkgs without declaring its specific version i.e., just use pythonPackages.numpy is the best way to use nix for dependency management?

Re: Freezing Python’s Dependency Hell

#128
post #94

Use a fresh virtualenv for each project As a form of version pinning, this locks in old versions and creates technical debt. A few years downstream, you're locked into library modules no longer supported and years behind in bug fixes.

The joy of not having to deal with broken production builds when dependencies change under your feet is well worth the "technical debt" in my opinion. Reproducible builds are valuable in their own right.

Re: Freezing Python’s Dependency Hell

#129
post #68

Earlier quoted context omitted.

Not OP, but we would like to move away from it as well. - Breaking behavior between minor versions ( https://github.com/conda/conda/issues/7290 ) - Environments not actually being isolated ( https://github.com/conda/conda/issues/448 ) - Can't create environments in long paths ( https://github.com/conda/constructor/issues/156 ) Those are just a few I can remember. We unfortunately have not found a strong replacement.

> Breaking behavior between minor versions See https://github.com/conda/conda/issues/7248 for where conda intends to head in the future on the environment.yml issue. > Environments not actually being isolated That's actually a really sticky issue, and one that's more about the python interpreter itself rather than anything conda is doing. More recent discussion at https://github.com/conda/conda/issues/7173 . Yes, we…

Quick question - has there been an attempt to make conda and official PEP and make conda-forge part of the Python Foundation (instead of a private company).

I'm trying to figure out why is all this new manylinux PEP stuff "inspired by conda" and is not actually conda.

Is this a situation like grsecurity vs Torvalds? How does the situation change now that BDFL is gone ?

Re: Freezing Python’s Dependency Hell

#130

Version pinning is technical debt and a fool's errand. New versions will always come out and your new development is confined to what once worked. You need to keep testing with current versions to see what will break when you upgrade and fix it as soon as possible so as to minimize the odds of a big breaking change. It may keep your environment stable for some time, but that stability is an illusion because the whole…

New versions will always come out, but it's not my job to test all of them. I'd rather consciously decide when I can afford to pay off the debt.
Post reply on HN