Live data from Hacker News

Ask HN: Why does every package+module system become a Rube Goldberg machine?

news.ycombinator.com

111–120 of 222 posts

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#111
post #5

At least in Python's case, I think people are much too hard on the packaging system. I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within. Following from this, I think that most back-end applications should try solve their messy runtime environment issues with some containerization. Java/C(++) however...

> I don't think anyone has significant issues with pip and boy would you be wrong. I just tried to install `datasette-scraper` on my system. This didn't work because it would consistently pick up outdated versions of Django and `more-itertools` from the system's Python site-packages directory. By the way I learned to (1) install an updated version of Python that has the advantage of not being the one that the system…

> no 'virtual-env' or some such!

I'm pretty sure you're manually doing the same thing virtualenv (venv in python3) does for you...

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#113
post #103
post #45

Earlier quoted context omitted.

2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in c…

Debian packages can have pre/postinstall scripts that mutate the overall state of the system in arbitrary ways. This is both (a) necessary to make some thing work and (b) ruins the theoretical model.

Those dirty practicalities breaking my perfect mental model again!

(This is a huge issue in our field in all sorts of ways, and a constant source of both pain and compromise)

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#114
post #103
post #45

Earlier quoted context omitted.

2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in c…

Debian packages can have pre/postinstall scripts that mutate the overall state of the system in arbitrary ways. This is both (a) necessary to make some thing work and (b) ruins the theoretical model.

> This is ... necessary to make some thing work

Things work in NixOS without the need of any of that. So, no it is not necessary.

It's a need created purely from the other directives of the Debian organization.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#115
Pythoners have themselves to blame.

The worst thing that happened to Python was that distributions like Red Hat not only shipped Python as an rpm but shipped system scripts in Python. This made it practically impossible to upgrade the system Python without breaking scripts and is fundamentally incompatible with how Python packaging works. (You are just one 'sudo pip install' from breaking your system)

Like people who are impressed with ChatGPT (wow that answer seemed so confident even if it is wrong), Pythoners systematically mistake footguns for solutions. For instance, use 'pip install --user' and now the package you've included is visible when you type 'python', 'python3', use a venv, conda, whatever. I've had people tell me that having a 'python3' is the best idea anyone's had since Solomon proposed cutting a baby in half, but if you stop and think you realize that pretty soon you have 'python3', 'python3.5' (broken), 'python3.6' (maybe broken), 'python3.7', 'python3.8', 'python3.9', ...

I would say venv's really solve the problem with the exception that you can currently trash all your venv's with 'pip install --user'.

The trouble is that people who are bothered by this stop using Python, the people who are still using Python are people who have difficulty perceiving the nature of the problem never mind that there is a problem.

The algorithm used by pip to resolve packages is not sound. It goes off trying to install things and can only find out late in the game that it installed a package which is not compatible with another package. This can be solidly blamed on the egg package format which can only determine the dependencies of a package when it tries to install it by running a setup.py. This is nice because it can adapt to the environment dynamically (say add a timezone db on windows) but it is not like Java's maven that knows it has a consistent solution before it downloads the JARs.

I'm pretty sure you could make a Python package manager that works like Maven if it only installed wheels because you can read the directory of a wheel (a ZIP file) with one or two http range requests and then read the dependency data with another http range request. Poetry doesn't quite do this. Poetry does a lot better than pip does, but whatever it does takes a really, really, really, long time.

Poetry might be the best thing going but it has the fundamental flaw of trying to be a 95% or 98% solution which, to the Pythoner, sounds like a good idea. The trouble is that writing programs that are 98% is to computer science what Bigfoot is to zoology or ESP is to psychology. The gap between that and 100% is not 2% but more like 200% because figuring out exactly what is wrong with the conceptual model and working around it in a hard case is at best like pushing a bubble under a rug. Imagine how much trouble you could get into with a 98% correct sort or binary search algorithm... It's a place where professional developers just don't want to go.

Java has the advantage of extreme xenophobia and bad feelings all around that mean that: (1) people don't expect to link very much C code into Java, and (2) people don't feel entitled to use the Java bundled with their Linux system and have it really work. Because of (1), Java code tends to be self-sufficient and avoids a whole sector of 'dependency hell' involving .so and .dll files. Because of (2) people feel both responsible and empowered to install a JDK that works unlike the typical who Pythoner doesn't.

Don't think that docker helps in any way, what I found what that Docker accelerates the ability of Pythoner to find pythons that are broken in odd ways, configured with strange default character sets and such.

Javascript has the nice feature that file A can import from package version B and file C can import from package version D so that the 'diamond dependency' problems that are so terrible in Java (Like that bug in Guava post-13 that broke HDFS) and still problematic in Python rarely produce problems. (You only have problems if an object from A migrates to C and it gets used in incompatible way, contrast that to Java where probably the classloader blows up)

I was wishing over the weekend that Python had something like that because once more I have been building and packaging machine learning models for inference in Python and it would be really sweet to pack up a model that uses scikit-learn or pytorch or whatever into a Python package and then load it into a web server or a batch job. It's totally practical to do that, using joblib to unpickle a few MB of code. Once you end up needing two different versions of pytorch though, you are really screwed.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#116
post #78
post #45

Earlier quoted context omitted.

2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in c…

> are we thinking about Debian packaging in completely different ways? Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it woul…

> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally"

apt-get --build source package_name

It's unlikely to be fully repeatable to the point of producing an identical binary, but it's also not some esoteric set of procedures.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#117
There is no real theory because it's a solved problem--packages and dependencies form a DAG or graph and it's just an ordering and traversal problem that's easily solved.

The pain and fragmentation you're mentioning is that everyone has different opinions about how they want to configure/bundle/organize code and package metadata. That's really the only core difference between deb/rpm, npm/yarn, pip/poetry/conda/, handmade makefile/cmake/autotools/etc. People just had different ideas about how they wanted to do things at the periphery. At their core all of those tools are just simple DAG walkers.

Basically, you're thinking it's a technical problem when in reality it's just a social issue. Some people prefer something one way, others prefer it the other way--there is no consensus (and likely never will be, people will always be building tools to do things their way).

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#118
Because anyone can write a mediocre "gets the immediate problem solved" type of package manager, while designing and implementing a programming language that someone else would want to use requires way more knowledge and skill.

Eventually the shortcomings of the package manager will become more evident, but at first it will seem to work and solve some pain points programmer have.

Something similar goes for build systems.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#119
post #5

At least in Python's case, I think people are much too hard on the packaging system. I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within. Following from this, I think that most back-end applications should try solve their messy runtime environment issues with some containerization. Java/C(++) however...

Everybody uses PIP, but the Python devs (well single BDFL) refuse to accept this and standardize the package manager. As a result, if you want to make a package, you must support everything.

Also, PIP can't evolve, because nobody will make their packages even more complex to support PIP only idiosyncrasies. It doesn't matter that everybody uses PIP, you have to spend 90% of your time dealing with other package managers, so they are the ones on your mind all the time.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#120
post #44
post #31

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager. For this to work at scale we'd…

Yes! I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.

Did you compare nix to npm? Is there a good comparison out there?
Post reply on HN