Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why does every package+module system become a Rube Goldberg machine?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 222 posts

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#71

Post by simiones »

pjmlp wrote: post #70
simiones wrote: post #55

Earlier quoted context omitted.

Modules typically refers to a unit of code that packages together several types and functions (and/or other top-level concepts in your language, e.g. macros, templates, metaclasses, namespaces, packages) with an explicitly declared public interface, while still allowing these types and functions to have additional internal APIs that are not exposed. A module has to declare what other modules it depends on explicitly.…

> C++ breaks C's module system because of its heavy reliance on templates, which have to be entirely declared in the public API and can't have any internal API of their own; and because the compiler needs to know the total size of a class, including the size of its private members, to compile any code which references objects of that class. Other languages with generics and modules do just fine by having the required…

Sure, I should have said "C++ before the modules feature". I was thinking more to explain why the modules feature was required at all (when it's not really required too much for C).

Also, C# Assemblies solved this problem for generics too, for a more mainstream example.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#72

Post by simiones »

ChrisRackauckas wrote: post #63
simiones wrote: post #56

Earlier quoted context omitted.

There will be a mess of dependencies of old projects that depend of projects that have since moved off GitHub, there will be name clashes between GitHub projects and BitBucket projects etc. These ecosystems are very much dependent on GitHub, and if/when anything permanent changes, it will be a mess to move off of it.

> there will be name clashes between GitHub projects and BitBucket projects etc. The package manager requires a canonical URL for the git repo. The location of dependencies doesn't matter, for example Github repos with Bitbucket dependencies work fine. The FastLapack packages lived on BitBucket at one point, and then migrated to Github, and that happened without a hitch to upstream packages like LinearSolve.jl or Dif…

I was mostly speculating, and thinking more of how Go does Github dependencies. I am happy to hear that Cargo doesn't have the same sort of issues.

I really should have looked into it more before commenting, especially since I knew Cargo is actually a well-designed and considered package system.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#73

Post by [deleted] »

ChrisRackauckas wrote: post #63
simiones wrote: post #56

Earlier quoted context omitted.

There will be a mess of dependencies of old projects that depend of projects that have since moved off GitHub, there will be name clashes between GitHub projects and BitBucket projects etc. These ecosystems are very much dependent on GitHub, and if/when anything permanent changes, it will be a mess to move off of it.

> there will be name clashes between GitHub projects and BitBucket projects etc. The package manager requires a canonical URL for the git repo. The location of dependencies doesn't matter, for example Github repos with Bitbucket dependencies work fine. The FastLapack packages lived on BitBucket at one point, and then migrated to Github, and that happened without a hitch to upstream packages like LinearSolve.jl or Dif…

[deleted]

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#74

Post by staunton »

simiones wrote: post #72
ChrisRackauckas wrote: post #63

Earlier quoted context omitted.

> there will be name clashes between GitHub projects and BitBucket projects etc. The package manager requires a canonical URL for the git repo. The location of dependencies doesn't matter, for example Github repos with Bitbucket dependencies work fine. The FastLapack packages lived on BitBucket at one point, and then migrated to Github, and that happened without a hitch to upstream packages like LinearSolve.jl or Dif…

I was mostly speculating, and thinking more of how Go does Github dependencies. I am happy to hear that Cargo doesn't have the same sort of issues. I really should have looked into it more before commenting, especially since I knew Cargo is actually a well-designed and considered package system.

Just to make sure there are no misunderstandings. ChrisRackauckas was talking about Julia's package manager, (which is called Pkg, if you need to name it, although I would just call it "the package manager"). Cargo is the Rust package manager.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#75

Post by denton-scratch »

nicoburns wrote: post #30
denton-scratch wrote: post #18

It annoys me that every gee-whizz new language needs to have it's own package-management system. There's no reason why a package-management system needs to be language-specific; dependencies are often cross-language. Hell, even some blocks of code contain more than one language. The package-management system is responsible for deploying packages. The way a package is deployed should depend on the operating environmen…

> Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk. Do you mean github? Git is open source and one of the few pieces of software that works in a truly distributed fashion.

You're right; I meant github. Sorry.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#76

Post by gwd »

tsimionescu wrote: post #46
jakewins wrote: post #6

Have you used Go or Rust package ecosystems? My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape. Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.

Go package management is a mess of hacks. It doesn't even have a repository, instead relying on source control systems to do the actual work, with special hacks for each source control system to define the artifacts that can be downloaded (e.g. using Git tags with some magic format, or Perforce commit Metadata). It requires you to physically move all code to a new folder in your version control if you want to increas…

> It relies on DNS for package identity.

The flip side of this is that it never has to worry about naming collisions or namespacing: Your public package name must be a URL you control.

Additionally, there is no requirement for a centralized package facility to be run. The Golang project is currently running pkg.go.dev, but that's only been in the last few years; and if they decided to get rid of it, it wouldn't significantly impact the development environment.

Finally, the current system makes "typo-squatting attacks" harder to do. Consider the popular golang package github.com/mattn/go-sqlite3. The only way to "typosquat" the package is to typosquat somewhere up the dependency tree; e.g., by creating github.com/matn/go-sqlite3 or something. You can't typosquat github.com/mattn/sqlite3, or github.com/mattn/go-sqlite, because you don't own those namespaces; whereas with non-DNS-based package systems, the package would be called `go-sqlite3`, and `sqlite3` or `go-sqlite` would be much easier to typosquat.

All those things I find really valuable; and honestly it's something I wish the Rust ecosystem had picked up.

> It requires users of your code to update all of their import statements throughout their code whenever you move your hosting.

This is a necessary cost of the item above. It can be somewhat annoying, but I believe this can be done with a one-line change to the go.mod. I'd much rather occasionally deal with this.

> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number.

And the benefit of this is that legacy code will continue to compile into the future. I do tend to find this annoying, but it was explicit trade-off that was decided back when they were developing their packaging system.

Packaging is a hard problem, with lots of trade-offs; I think Go has done a pretty good job.

One way in which Go and Rust have it easier than Python or Node is that the former only have to deal with developers; the latter have to deal with both developers and users, whose requirements are often at odds with one another.

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#78

Post by tazjin »

pastage wrote: post #45
tazjin wrote: post #31

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager. For this to work at scale we'd…

2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in c…

> are we thinking about Debian packaging in completely different ways?

Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number).

> Something that even Nix struggles with

Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course.

> I wonder what you consider the goal of a package system

I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed.

Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#79

Post by tomcam »

pjmlp wrote: post #20
jakewins wrote: post #6

Have you used Go or Rust package ecosystems? My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape. Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.

Doesn't look like it, see module drama in Go, while Rust is having an npm like ecosystem of tiny crates. Plus none of them handle binary library distribution as some of the packing models that came before them.

Can you give me a one-liner about module drama in Go?

Re: Ask HN: Why does every package+module system become a Rube Goldberg machine?

#80

Post by eurasiantiger »

xiphias2 wrote: post #23
jakewins wrote: post #6

Have you used Go or Rust package ecosystems? My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape. Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.

Npm has lots of not understandable mistakes: it should already be as good with ES6+TS as other systems. Import should just work everywhere. We should have left Commonjs a long time ago, while keeping backwards compatibility. At the same time what I see with the node+npm system is that everything is just ,,it just doesn't work by default''. Having 10 other package managers doesn't work either, they are faster, but don…

Just use .mjs extension for your code, add extensions to your imports and everything does just work.
Post reply on HN