Ask HN: Why does every package+module system become a Rube Goldberg machine?

A programming language has a "core language" plus a package/module system. In each successful language, the core language is neat-and-tidy, but the package/module system is a Rube Goldberg machine. See JavaScript/TypeScript, Python, or C/C++.

Lots of brain cycles are spent on "programming language theory". We've roughly figured out the primitives required to express real-world computation.

In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies. As a result, we keep building new variants and features, until we end up with , require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...

Is it just a "law of software" that this must happen to any successful language? Or are there examples of where this it has not happened, and what can we learn from them? Is there a "theory of package management", or a "lambda calculus of package management" out there?

Julia and Rust seem to have package systems that are fine and manageable. I think these are really just major problems in Javascript, Python, and C/C++ (exactly the languages you mention) because the kind of widespread OSS code sharing just didn't exist to the extent it does today back when those languages were designed. People shared code through email, but didn't expect one button to pull in 200 Github repositories, and thus weren't built with the expectations required to make that be stable.

Back when those languages were designed, you'd manually download the few modules you need, if you downloaded any packages at all. In C you'd normally build your own world, since it came before the www times, and C++ kind of inherited that. But languages which came out later decided that we now live in a world where most of the code that is executed is packages, most likely packages which live on Github. So Julia and Rust build this into the language. Julia in particular with the Project.toml and Manifest.jl for fully reproducing environments, its package manager simply uses git and lets you grab the full repository with `]dev packagename`, its package registry system lets you extend with private package worlds.

I think the issue is that dependencies are central, so you can never remove old package systems because if that's where the old (and rarely updated) dependencies live, then you need to keep it around. But for dependencies to work well, you need all dependencies to be resolved using the same package system. So package systems don't tend to move very fast in any language, whatever you had early has too much momentum.

At least in Python's case, I think people are much too hard on the packaging system. I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within.

Following from this, I think that most back-end applications should try solve their messy runtime environment issues with some containerization. Java/C(++) however...

> See JavaScript/TypeScript, Python, or C/C++.

I feel like you've picked the 3 worst ecosystems as an example.

Java has a wonderful ecosystem. You can use Maven/Gradle/Bazel as a dependency manager/build tool. All support the Maven repository format. Packages are published as jars and easily consumed. Of course you can still end up in dependency hell and you should take steps to mitigate those issues, but I'm quite happy with it.

> As a result, we keep building new variants and features, until we end up with , require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...

IMO the problem here is that there isn't a monopoly and its easy to swap in a replacement. JavaScript/Python developers are happy (or willing) to try something new, so new alternatives are created. Convincing a Java developer to try a new build system is difficult, so changes are made to existing systems rather than creating new ones.

Go and Rust are two other systems that work well and are (almost?) universally adopted. They come with the language and there is little reason to look around.

I just spent a few hours in npm/node/heroku hell, so... yeah. It's awful.