Writing a Package Manager • Hacker News Board

Re: Writing a Package Manager

#71

Post by patmorgan23 » Wed, Aug 23, 2023, 2:09 PM UTC

brabel wrote: post #52

zivkan wrote: post #40

Earlier quoted context omitted.

Depending on how important supply chain security is to your industry/company/team, validating the hash of every package is critical. If an attacker can manage an interception/man in the middle attack on your CI network, the hash check provides protection. If an attacker compromises the server you get packages from, having a hash provides protection as well. Automatically trusting the server's response to be correct,…

Your argument supports the idea of getting rid of the lock file and instead committing the hash in the original dependencies file - so that it's never an automatic process to update the hashes/versions.

Zigs package manager takes this approach.

https://zig.news/edyu/zig-package-manager-wtf-is-zon-558e

Re: Writing a Package Manager

#72

Post by dr-detroit » Wed, Aug 23, 2023, 2:14 PM UTC

brabel wrote: post #25

The author added a lockfile without understanding why they exist. A lockfile is meant to "freeze" dependency version resolution when package authors can specify dependencies on other packages using version ranges... it also "freezes" choices of transitive packages' versions when different packages depend on the same one, but with different versions. They chose to not handle package dependencies at all, and I believe…

[dead]

Re: Writing a Package Manager

#73

Post by jbverschoor » Wed, Aug 23, 2023, 2:27 PM UTC

grumblingdev wrote: post #58

david2ndaccount wrote: post #19

Honestly pretty strange to write a package manager for sqlite and to use directories + json files to store the data instead of sqlite.

SQL not so good for tree structures. Recursive CTEs are pretty gnarly. Graphdb would be ideal.

SQLite is a pretty darn good solution for any file(set) that needs atomic / consistent writes

Re: Writing a Package Manager

#74

Post by nalgeon » Wed, Aug 23, 2023, 2:39 PM UTC

brabel wrote: post #25

The author added a lockfile without understanding why they exist. A lockfile is meant to "freeze" dependency version resolution when package authors can specify dependencies on other packages using version ranges... it also "freezes" choices of transitive packages' versions when different packages depend on the same one, but with different versions. They chose to not handle package dependencies at all, and I believe…

There is quite a difference between "the author added a lockfile without understanding why they exist" and "I really don't see why they added a lockfile".

It's better to start with the latter if you seek to understand, not attack.

Re: Writing a Package Manager

#75

Post by hgs3 » Wed, Aug 23, 2023, 2:39 PM UTC

planede wrote: post #32

So, how does it handle diamonds in the dependency graph? How does it handle version compatibility? IMO these are the most interesting aspects of a package manager, other stuff are implementation detail. Although there is a mention for semver at the end: > Decide what versioning scheme to use (Probably semver, or something like it/enhancing it with a total order). I wonder if total order is appropriate (assuming the r…

Agreed. Version resolution is the interesting problem.

Most package managers use a SAT solver to resolve dependencies. The Dart team has a detailed write up on their SAT-based approach which is worth a read [1]. For contrast, Russ Cox presents an algorithm that doesn't use a SAT solver (intended for Go) [2].

[1] https://github.com/dart-lang/pub/blob/master/doc/solver.md

[2] https://research.swtch.com/vgo-mvs

Re: Writing a Package Manager

#76

Post by nalgeon » Wed, Aug 23, 2023, 2:41 PM UTC

david2ndaccount wrote: post #19

Honestly pretty strange to write a package manager for sqlite and to use directories + json files to store the data instead of sqlite.

A couple of reasons:

1) I wanted human readable specs and lockfiles.

2) I didn't want to introduce a dependency (working with SQLite in Go is pretty ugly, it either requires GCC or a ton of other dependencies for a pure Go implementation).

Re: Writing a Package Manager

#77

Post by nonameiguess » Wed, Aug 23, 2023, 3:14 PM UTC

planede wrote: post #32

So, how does it handle diamonds in the dependency graph? How does it handle version compatibility? IMO these are the most interesting aspects of a package manager, other stuff are implementation detail. Although there is a mention for semver at the end: > Decide what versioning scheme to use (Probably semver, or something like it/enhancing it with a total order). I wonder if total order is appropriate (assuming the r…

I could be wrong here, but given what sqlite's documentation includes and the fact his spec files don't even include dependencies, I believe sqlite extensions don't depend upon anything except sqlite itself (he says here "most" don't).

You'd need to ensure the extensions and sqlite were both compiled against the same libc, but I don't see what this package manager can do about this given the metadata doesn't seem to be available in these Github releases. In fact, going to the repo for the example in his spec file, the reason for the release was to downgrade the build host OS from Ubuntu 22.04 to 20.04 to resolve an issue with some user not being able to run this because of a missing GLIBC symbol.

This is an underappreciated problem and why actual distros include everything. If you're going to distribute binary files, you need to ensure they're compatible in many more ways than just the architecture and kernel matching. A complete package manager would include a build system and public registry with builds matching a complete machine triplet (as in, what you'd get by running gcc -dumpmachine). If you consume upstream Github releases as this is doing, they may not have that level of granularity, and in fact, we can see they do not.

Re: Writing a Package Manager

#78

Post by crabbone » Wed, Aug 23, 2023, 3:20 PM UTC

BearhatBeer wrote: post #5

I think some day rather than the current paradigm, even including declarative package managers and environments and distros, the future will be per-user and even per-app chrooting or jails, or something similar. Apple already uses something like this today. Many people who are smart about information security have one login for shopping and banking and bill pay, one for their business, and one for cruising the web or…

This idea is as confused as people who claim that they "don't need package manager because they have Docker".

So what if you will have containers / jails? -- You still need to install multiple components into the same container / jail... because you need them to work together. It's not solving the problem at all. Of course, containers are useful, but not for the purpose of solving installation of software comprised of multiple components problem.

Re: Writing a Package Manager

#79

Post by crabbone » Wed, Aug 23, 2023, 3:23 PM UTC

planede wrote: post #32

So, how does it handle diamonds in the dependency graph? How does it handle version compatibility? IMO these are the most interesting aspects of a package manager, other stuff are implementation detail. Although there is a mention for semver at the end: > Decide what versioning scheme to use (Probably semver, or something like it/enhancing it with a total order). I wonder if total order is appropriate (assuming the r…

Doesn't the article say that it doesn't deal with dependencies at all?

Precisely, I wouldn't call this a "package manager", it's a "package installer" (same as pip). Managing implies some form of interaction after components are installed (esp. ensuring that all installed components can work together). This project doesn't seem to manage anything, it's more of a fire-and-forget kind of thing.

Re: Writing a Package Manager

#80

Post by nonameiguess » Wed, Aug 23, 2023, 3:25 PM UTC

forrestthewoods wrote: post #4

Wonderful post! I've written a couple of packager manager like things over the years. Although they've all been internal-only which gives some flexibility. > I like the idea of allowing both project and global scope Yikes! Hard no from me. Globals are pure evil. Avoid like the plague. Environment variables too. Kill them all with fire. > what if the user wants to reinstall the packages on another machine or CI server…

> I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control!

These are binary dependencies. You're checking shared object files into source control? Note that he's also doing this to add functionality to a binary installation of sqlite, which is presumably running somewhere like /usr/bin/sqlite. This isn't a custom application he's developing. The extensions are in his home directory. "Checking them into version control" would entail making his entire filesystem a Git repo. If you're willing to pay whatever IBM charges these days for ClearCase, something like that might actually work. Regular source control like Git, though? I don't think so.