Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why does every package+module system become a Rube Goldberg machine?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 222 posts

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#171

Post by anderskaseorg »

0x0 wrote: post #170
anderskaseorg wrote: post #168

Earlier quoted context omitted.

Even in the best case where Debian satisfies all the reproducibility goals it sets for itself (i.e. the ~96% at https://tests.reproducible-builds.org/debian/reproducible.ht... becomes 100%), each Debian package is still only a reproducible function of the current versions of its build dependencies at the time it was built. Upgrading those dependencies can change the build output, and this is still normal and expected…

Well of course it snakes back, and likely beyond debian's inception, your initial c compiler has to come from somewhere.

Right. Nix bootstraps from a fixed collection of seed binaries, and everything in any given commit of nixpkgs is derived from the seed binaries and versions of software that exist in that commit of nixpkgs. It doesn’t solve the “trusting trust” problem with those seed binaries, but it does solve the “arbitrarily long chains” problem.

(The work at https://www.gnu.org/software/mes/ aims to minimize the needed binary seed to a human-auditable size, and should be applicable to Nix and Guix.)

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#172

Post by whateveracct »

ttyprintk wrote: post #2

It sounds like you’re familiar with the status quo, so let’s look at this as a matter of requirements: - Cross-platform - Wraps another packaging format - Lowest disk space - Dependency version solving - An OS can run make changes to its own low-level state using its own tools - User can configure when and what are installed Even Nix, which has its own language and isolation model, can’t meet all these.

Nix almost does meet those though.

It's cross-platform to some degree. If Windows is a requirement for this, I see why it doesn't qualify.

Nix wraps other packaging formats all the time. There's endless x2nix converters.

In a way, Nix is "lowest disk space" - by nature of lazy evaluation, Nix closures are always exactly what they need. Now, they may "actually" need less and are holding onto more due to how packages are written by people. And you do need to GC.

Dependency version solving can actually be punted to other tools. In Haskell, Nix uses a combination of cabal and pkg-config to handle Haskell + C deps.

NixOS can swap out the kernel along with anything else with a line or two in your config.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#173

Post by whateveracct »

hiljusti wrote: post #160
noloblo wrote: post #82

What about haskell cabal and stack

I appreciate Hackage and its lack of garbage, although I think the exclusivity can't scale. I also like Haskell's module system All other aspects of Haskell package management feel unfinished, like "an exercise left to the reader." The lack of polish is astounding. See also: "Why I Don't Code in Haskell Anymore" ( https://www.youtube.com/watch?v=SPwnfSmyAGI ) from TsodingDaily

You don't have to use Hackage.

What part is unfinished? Just docs and the verbose CLI? It solves for versions, you can declare native dependencies in various ways, it has multiple solid integrations with Nix (which in turn automatically resolves both Haskell and native deps for you).

Also that video is not a compelling source. That guy is just wrong lol. Or, rather, it's just his opinion that seems to be based on the 5-10 years ago state of Haskell.

I'm an engineer (professionally - in Haskell - for years), and I don't find anything about the packaging or "engineering" painful.

If we're talking engineering, the optimizing compiler + RTS are extremely impressive.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#174

Post by Someone »

If you let everyone and their dog create and maintain packages and package dependencies and leave it to them to declare forward/backward compatibility between versions of packages, and your package manager ends up being popular, I think it’s unavoidable that you end up with a mess.

Foo may not be updated for Bar 2.0, while the Baz you use states it needs it, Bar 3.0 may incorrectly declare to be a 100% stand-in for anything between Bar 1.5 and Bar 3.0, Baz and Qux may contain similar functions that would better be part of their shared dependency Quux, etc.

Tooling can make navigating that mess more or less difficult, but the mess still is there.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#175

Post by lamontcg »

jmull wrote: post #147
qbasic_forever wrote: post #117

There is no real theory because it's a solved problem--packages and dependencies form a DAG or graph and it's just an ordering and traversal problem that's easily solved. The pain and fragmentation you're mentioning is that everyone has different opinions about how they want to configure/bundle/organize code and package metadata. That's really the only core difference between deb/rpm, npm/yarn, pip/poetry/conda/ , ha…

Well, not DAG, because the A is for acyclic, and dependency graphs can definitely contain cycles.

They really shouldn't, and some dependency systems like rubygems/bundler after the molinillo switch ban them entirely.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#176

Post by pxc »

goodpoint wrote: post #159
tazjin wrote: post #78

Earlier quoted context omitted.

> are we thinking about Debian packaging in completely different ways? Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it woul…

> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.

> It's been there for 15 or 20 years.

Has it? What I remember being the standard practice for customizing packages was

  apt-get source && sudo apt-get build-dep && vim  && debuild -b
fewer than 15 years ago. And it's not much like what the parent poster has in mind. It modifies the global state of your system, it requires multiple steps, it produces a different artifact than you might get in the repos at any given time, it leaves things in your CWD on your filesystem owned by your current user, whatever.

(Some of that may also apply to `apt-get source --build`, idk.)

With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.

I encourage you to actually try Nix or Guix, including writing packages for them, and including adding a patch or overriding a build flags in an existing package, to see how the model is really different.

> Also Debian has been doing reproducible builds before Nix.

What Debian does for reproducible builds and how Nix embodies reproducibility are qualitatively different. If you understood both, you'd understand why this is a silly point to raise.

(Debian's work on reproducible builds is great. Every distro benefits from most of it, and everyone in the Nix community is grateful for it.)

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#177

Post by pxc »

hiljusti wrote: post #152
marcosdumay wrote: post #114

Earlier quoted context omitted.

> This is ... necessary to make some thing work Things work in NixOS without the need of any of that. So, no it is not necessary. It's a need created purely from the other directives of the Debian organization.

Not necessary, but certainly it's easier to implement and get stuff done. It starts to break down and show its cracks the larger and larger it gets, though. The difference between almost all Linux or BSD packaging systems and nixpkgs is like the difference between dynamic and compile-time-type-checked programming languages.

You can have a much more conventional package manager without pre- or post-install hooks. One ex-Debian developer created such a package manager as a research project, presumably to prove a point to other Debian people with the same head-in-the-sand attitude about the state of Debian's package tooling.

It doesn't require the same kind of patching as Nix does (although sometimes patching does help avoid requiring those scripts).

Check it out: https://distr1.org/

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#178

Post by pxc »

Nursie wrote: post #113
pjc50 wrote: post #103

Earlier quoted context omitted.

Debian packages can have pre/postinstall scripts that mutate the overall state of the system in arbitrary ways. This is both (a) necessary to make some thing work and (b) ruins the theoretical model.

Those dirty practicalities breaking my perfect mental model again! (This is a huge issue in our field in all sorts of ways, and a constant source of both pain and compromise)

Alternatively: that pesky lack of discipline obstructing my optimizations and features again.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#179

Post by pxc »

folex wrote: post #44
tazjin wrote: post #31

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager. For this to work at scale we'd…

Yes! I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.

Check out Denxi. Might be the closest thing to what you're looking for today, and is informed by the state of the art in package management without totally imposing strict discipline on all packages.

https://docs.racket-lang.org/denxi-guide/index.html

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#180

Post by pxc »

tazjin wrote: post #134
puffoflogic wrote: post #124

Earlier quoted context omitted.

Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.

[flagged]

Pretend I asked the nice version of this, for the benefit of other readers?

I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?

Post reply on HN