The fundamental problem of programming language package management

> To learn Nix you need to learn both how to write shell scripts and how to write Nix expressions. How is that better than just learning Scheme, which is very trivial to learn the basics - and for most packages, you don't really need to learn much because you can reference other packages - it's really just like a configuration file.

Except that as a Linux admin, you probably already know how to write shell scripts anyway, and you'd be rather hard-pressed to manage a Linux system while never once having to write or read one. There's also a lot more information on writing shell scripts than Guile scripts for various common purposes.

> scheme is the only language you'll need to be able to fully drive your OS.

With a different DSL for each use, meaning you have to learn the restrictions of each DSL anyway.

> You can do anything from the shell too (which nixpkg can invoke) - you can even invoke guile from a shell script.

That's true, although in Nix's case, the derivations map directly to the source, as Nix code itself can't call out to anything (only return derivations which might) - and in theory, the build could then happen in a sandbox, making it even more likely that the result would be the same.

We might not have reproducible builds yet, but Nix is closer to having them than Guix if somebody wanted to make a research project out of it.

Does OPAM work on Windows? I've read it doesn't which has kept me from further investigations.

From a dumb-user perspective, I feel like most of the effort of package managers is spent on resolving these fundamentally incompatable philosophical dilemmas instead of common-denominator solvable problems like:

* Quality and Trust mechanisms. If there are 14 different postgres clients, which do I choose?

* Package Metadata management. Where can I send bug reports? Who is the maintainer? How can I contact someone? Is there an IRC channel?

* Documentation and Function/Class Metadata. Why should I go to the Github README for one package, and to a random domain for another package?

* Linking compile and runtime error messages to documentation or bug reports. Why is google still the best way to track down the cause of an obscure error message?

* Source data linking and code reviews. I should be able to type in a module/namespace qualified function name and view the source without having to scour a git repository. I should also be able to comment directly on that source in a way that is publicly visible or privately visible.

The situation with packages and dependency hell today is horrendous, particularly if you work in a highly dynamic environment like web development.

I want to illustrate this with a detailed example of something I did just the other day, when I set up the structure for a new single page web application. Bear with me, this is leading up to the point at the end of this post.

To build the front-end, I wanted to use these four tools:

- jQuery (a JavaScript library)

- Knockout (another JavaScript library)

- SASS (a preprocessor to generate CSS)

- Jasmine (a JavaScript library/test framework)

Notice that each of these directly affects how I write my code. You can install any of them quite happily on its own, with no dependencies on any other tool or library. They are all actively maintained, but if what you’ve got works and does what you need then generally there is no need to update them to newer versions all the time either. In short, they are excellent tools: they do a useful job so I don’t have to reinvent the wheel, and they are stable and dependable.

In contrast, I’m pretty cynical about a lot of the bloated tools and frameworks and dependencies in today’s web development industry, but after watching a video[1] by Steven Sanderson (the creator of Knockout) where he set up all kinds of goodies for a large single page application in just a few minutes, I wondered if I was getting left behind and thought I’d force myself to do things the trendy way.

About five hours later, I had installed or reinstalled:

- 2 programming languages (Node and Ruby)

- 3 package managers (npm with Node, gem with Ruby, and Bower)

- 1 scaffolding tool (Yeoman) and various “generator” packages

- 2 tools that exist only to run other software (Gulp to run the development tasks, Karma to run the test suite) and numerous additional packages for each of these so they know how to interact with everything else

- 3 different copies of the same library (RequireJS) within my single project’s source tree, one installed via npm and two more via Bower, just to use something resembling modular design in JavaScript.

And this lot in turn made some undeclared assumptions about other things that would be installed on my system, such as an entire Microsoft Visual C++ compiler set-up. (Did I mention I’m running on Windows?)

I discovered a number of complete failures along the way. Perhaps the worst was what caused me to completely uninstall my existing copy of Node and npm — which I’d only installed about three months earlier — because the scaffolding tool whose only purpose is to automate the hassle of installing lots of packages and templates completely failed to install numerous packages and templates using my previous version of Node and npm, and npm itself whose only purpose is to install and update software couldn’t update Node and npm themselves on a Windows system.

Then I uninstalled and reinstalled Node/npm again, because it turns out that using 64-bit software on a 64-bit Windows system is silly, and using 32-bit Node/npm is much more widely compatible when its packages start borrowing your Visual C++ compiler to rebuild some dependencies for you. Once you’ve found the correct environment variable to set so it knows which version of VC++ you’ve actually got, that is.

I have absolutely no idea how this constitutes progress. It’s clear that many of these modern tools are only effective/efficient/useful at all on Linux platforms. It’s not clear that they would save significant time even then, compared to just downloading the latest release of the tools I actually wanted (there were only four of those, remember, or five if you count one instance of RequireJS).

And here’s the big irony of the whole situation. The only useful things these tools actually did, when all was said and done, were:

- Install a given package within the local directory tree for my project, with certain version constraints.

- Recursively install any dependent packages the same way.

That’s it. There is no more.

The only things we need to solve the current mess are standardised, cross-platform ways to:

- find authoritative package repostories and determine which packages they offer

- determine which platforms/operating systems are supported by each package

- determine the available version(s) of each package on each platform, which versions are compatible for client code, and what the breaking changes are between any given pair of versions

- indicate the package/version dependencies for a given package on each platform it supports

- install and update packages, either locally in a particular “virtual world” or (optionally!) globally to provide a default for the whole host system.

This requires each platform/operating system to support the concept of the virtual world, each platform/operating system to have a single package management tool for installing/updating/uninstalling, and each package’s project and each package repository to provide information about versions, compatibility and dependencies in a standard format.

As far as I can see, exactly none of this is harder than problems we are already solving numerous different ways. The only difference is that in my ideal world, the people who make the operating systems consider lightweight virtualisation to be a standard feature and provide a corresponding universal package manager as a standard part of the OS user interface, and everyone talks to each other and consolidates/standardises instead of always pushing to be first to reinvent another spoke in one of the wheels.

We built the Internet, the greatest communication and education tool in the history of the human race. Surely we can solve package management.

[1] http://blog.stevensanderson.com/2014/06/11/architecting-larg...

Sure we can. I imagine first we'll need the ISO to create a project to begin the standardization of software management. Then there'll probably be a few years of research to identify all the kinds of software, platforms they run on, interoperability issues, levels of interdependencies, release methodologies, configuration & deployment models, maintenance cycles, and expected use cases. Then the ISO can create an overly-complex standard that nobody wants to implement. Finally somebody will decide it's easier to just create smaller package managers for each kind of software and intended use case and write layers of glue to make them work together.

So now that we know what to do, the big question is: who's going to spend the next 5-10 years of their life on that project?