Ask HN: Why does every package+module system become a Rube Goldberg machine?

The concept of a programming language isn't what people use, but the concrete reality of specific implementations on specific hardware.

You could also say that a coffee cup and a donut aren't actually different because they're topologically the same... but the facts on the ground do actually matter.

Step 1: look at all the existing package+module systems, and realise they are all horrible Rube Goldberg machines, way too bloated for the simple thing you want to do

Step 2: write your own package+module system! It's lean, mean and just does the simple obvious thing your simple one-man project needs.

Step 3: your system hits the real world! Your system is used with projects that involve more than one person and grow organically, and that spread to more and more esoteric environments. The Rube Goldberg projects have Rube Goldberg aims and their target ecosystems have Rube Goldberg requirements while goalposts shift continuously with tight deadlines. You add features and toggles to your system to support these things.

Step 4: congratulations! You are the proud inventor and owner of yet another Rube Goldberg machine. It is now time to move on to better things; the system, meanwhile, is handed over to some committee and will only grow in complexity from here.

https://www.kiplingsociety.co.uk/poem/poems_palace.htm

The public interface of a module has to be explicit enough that a compiler for that language only needs the public interface description of that module to compile code that depends on it.

A good example of the most widely used and successful module system is C compiled libraries. The header files represent the public description of a library (or, for dynamic linking, the .so/.dll itself embeds this info as well), and a C compiler only requires these header files to be able to compile dependent code and later link the two. It is also impossible to access non-public members of the library from standard C (without resorting to direct memory access, of course).

Java packages don't have these properties because of two problems: there is no way to indicate dependencies between packages, and there is no standard way to have a bundle that corresponds to a package (since different .jar files can all load classes into the same package). The Java module system was created specifically to address these problems (and is still not fully adopted throughout the ecosystem).

C++ breaks C's module system because of its heavy reliance on templates, which have to be entirely declared in the public API and can't have any internal API of their own; and because the compiler needs to know the total size of a class, including the size of its private members, to compile any code which references objects of that class.

For Python, I'm honestly not sure if Python packages fit this definition of a module or not.

and boy would you be wrong. I just tried to install `datasette-scraper` on my system. This didn't work because it would consistently pick up outdated versions of Django and `more-itertools` from the system's Python site-packages directory.

By the way I learned to (1) install an updated version of Python that has the advantage of not being the one that the system uses; (2) do not use `pip` the executable but `python3.9 -m pip`; (3) use `pip` with argument `--target vendor` to install everything in a dedicated project-owned directory; (4) prefix my invocation of the software with `PYTHONPATH=./vendor` to add that to the import path; (5) add a `vendor/sitecustomize.py` file with the lines `import sys; sys.path.pop()` to remove the system's `site-packages` from the path.

That's a local installation of a Python package in five easy steps, no 'virtual-env' or some such! Pip is great! /s