Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why does every package+module system become a Rube Goldberg machine?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 222 posts

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#181

Post by pxc »

jmull wrote: post #147
qbasic_forever wrote: post #117

There is no real theory because it's a solved problem--packages and dependencies form a DAG or graph and it's just an ordering and traversal problem that's easily solved. The pain and fragmentation you're mentioning is that everyone has different opinions about how they want to configure/bundle/organize code and package metadata. That's really the only core difference between deb/rpm, npm/yarn, pip/poetry/conda/ , ha…

Well, not DAG, because the A is for acyclic, and dependency graphs can definitely contain cycles.

Yeah, that's a misfeature. It should definitely be a DAG. Which ecosystems 'support' that besides Node?

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#182

Post by pxc »

PaulHoule wrote: post #115

Pythoners have themselves to blame. The worst thing that happened to Python was that distributions like Red Hat not only shipped Python as an rpm but shipped system scripts in Python. This made it practically impossible to upgrade the system Python without breaking scripts and is fundamentally incompatible with how Python packaging works. (You are just one 'sudo pip install' from breaking your system) Like people who…

This is a very good rant, worth bookmarking to refer others to who are curious about packaging problems

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#183

Post by otabdeveloper4 »

puffoflogic wrote: post #124
tazjin wrote: post #31

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager. For this to work at scale we'd…

Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.

Nix is an umbrella for a stack of various solutions.

Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#184

Post by otabdeveloper4 »

pxc wrote: post #180
tazjin wrote: post #134

Earlier quoted context omitted.

[flagged]

Pretend I asked the nice version of this, for the benefit of other readers? I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?

Nix is capable of that, but you need a database mapping versions to VCS refs.

Nixpkgs doesn't go down that road because it would be unwieldy at their scale, but more specialized nix-based tools do.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#185

Post by puffoflogic »

otabdeveloper4 wrote: post #183
puffoflogic wrote: post #124

Earlier quoted context omitted.

Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.

Nix is an umbrella for a stack of various solutions. Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.

Under this abuse of terminology, a pocket calculator is capable of becoming a package/dependency manager. Or any other technology which conceivably could be put to work toward the package management problem, but has not yet been adapted to that need. I concede that these statements are "true", but they fail the relevancy test of communication.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#186

Post by pxc »

otabdeveloper4 wrote: post #184
pxc wrote: post #180

Earlier quoted context omitted.

Pretend I asked the nice version of this, for the benefit of other readers? I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?

Nix is capable of that, but you need a database mapping versions to VCS refs. Nixpkgs doesn't go down that road because it would be unwieldy at their scale, but more specialized nix-based tools do.

(I actually think Nixpkgs could and should go down this road, with a couple of caveats, for immense benefit. But I'm curious about what tazjin thinks here.)

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#187

Post by otabdeveloper4 »

puffoflogic wrote: post #185
otabdeveloper4 wrote: post #183

Earlier quoted context omitted.

Nix is an umbrella for a stack of various solutions. Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.

Under this abuse of terminology, a pocket calculator is capable of becoming a package/dependency manager. Or any other technology which conceivably could be put to work toward the package management problem, but has not yet been adapted to that need. I concede that these statements are "true", but they fail the relevancy test of communication.

Like I said in another comment, the problem is curating the giant database that maps version numbers to commit refs. Once you pass this roadblock, the tools Nix gives you make dependency resolution a quite a bit simpler problem to solve.

The next step is then realizing you actually don't need the useless legacy version numbering scheme at all, and that you wasted that effort for nothing.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#188

Post by jrockway »

briffle wrote: post #169
jrockway wrote: post #166

You mention Python and Node, which are programming language that unusually require end users to have the text of your program and all its dependencies on their own machine, and the languages store some parts of your program in /usr/lib and other parts of your program in your source directory. (npm does a little better here and at least puts the dependencies in your project directory.) Those constraints make developme…

if npm and pip are DARK AGES what does that mean perl's CPAN is?

I haven't used Perl for 15 years, but it was pretty miserable back then. Obviously I had my workflow and didn't run into too many problems, but I was certainly nervous about how I could share my work with other people. (Putting it into prod wasn't that bad. I don't know why, but it always went OK. CPANPLUS helped at the time.) I used to teach Perl trainings as a side job, and people would literally be in tears over @INC. It was bad.

I don't use Python much these days, but it's not as bad as Perl 15 years ago. I see blog posts like "how to set up the perfect Python dev environment with Docker" and it makes me very sad, but at least teams are getting their work done. The edge cases of Python packaging, though, are really really bad. For example, the C compiler that Python was compiled with (and C library; musl vs. glibc) affects installability of modules through pip. This really forces your Linux distribution to be the arbiter of what modules you can use, which is always too out of date for development. Also, the exact requirements (what are the sha256s and URLs of the code files that this code depends on) depends on architecture, compiler, etc. As far as I can tell, you have to run code to find the whole dependency tree. That is the fatal flaw I see with Python's packaging system.

I spent a lot of time trying to use Bazel to unify all the work my company does across languages. Python was the killer here. We do depend on a lot of C extensions, and I have a C toolchain built with Bazel (so that arm64 mac users can cross-compile to produce Linux releases; maybe a dumb requirement, but it would be too unpopular to discriminate against certain developer devices), but getting Python built with that toolchain, and using that Python to evaluate requirements.txt didn't work. (There are bugs open; they all end with "Python has to fix this".) As a result, you don't get compatible versions of, say, numpy with this setup. Dealbreaker. Partially Bazel's fault, partially Pyton's fault.

(It pained me to use Bazel for Go, but it did all work. While the workflow wasn't as nice as what Go has built in, easy things were hard and hard things were possible. I had working binaries within a few hours, buildable on any supported workstation type, with object and test caching in the cloud.)

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#190

Post by tremon »

pxc wrote: post #176
goodpoint wrote: post #159

Earlier quoted context omitted.

> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.

> It's been there for 15 or 20 years. Has it? What I remember being the standard practice for customizing packages was apt-get source && sudo apt-get build-dep && vim && debuild -b fewer than 15 years ago. And it's not much like what the parent poster has in mind. It modifies the global state of your system, it requires multiple steps, it produces a different artifact than you might get in the repos at any given time…

With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.

I consider this to be one of the main drawbacks of these systems -- it assumes that every machine is build-capable. I do not want compilers installed on my production servers (nor would I want my RPi to attempt to rebuild half the world because the package server was down). I'm not sure how Nix handles this, but I'm pretty sure that's not possible with Guix.

Post reply on HN