Nix as OS X Package Manager

Hi, I'm a NixOS/Nixpkgs comitter and package maintainer for plenty of stuff -- including X11/XQuartz for OSX.

Nope. They both do package management, but that's where the similarities end.

Homebrew packages are not patched to point at precisely the version of the stuff they were built against. For example, if you have program that dynamically links to openssl, that program is built with the expectation that the dynamic linker will be able to find e.g. libssl.dylib in either /usr/lib or /usr/local/lib (the specifics are harrier, but that's a decent approximation). If you later upgrade openssl and the ABI changes, you've now silently broken everything that uses it -- you'll now get a segfault at runtime (I've had this happen with openssl and many other libs on Homebrew, and is part of why I was (and still am) very excited about Nix).

When you compile a package with Homebrew, you're not guaranteed to end up with the same result as someone else, even if they have the same checkout of the formulae. Why? Well, each project's Makefile (or what have you) will try to run whatever's on $PATH, and poke around elsewhere on your system to e.g. automatically enable build flags (oh, luajit is installed? I guess I'll just go ahead and enable Lua scripting and link to it).

How does that compare with Nix?

Nix packages are patched so that their runtime dependencies (dynamic libraries, programs to be execed, shebang lines, etc) are locked down to the precise build that was specified as part of the package. The obvious implication here is that Nix packages can be trivially installed with differing versions of dependencies as necessary (back in the day, I wanted to play with both the Elixir langauge and the Riak DB, but they required two different major versions of Erlang. Unfortunately, I could only install one version, as the packages for both versions wanted to be placed in the exact same prefix and the filenames would overlap. This was on Ubuntu, but it applies equally to Homebrew).

When compiling a Nix package, I can rest assured that the build artifacts will be equivalent between two different machines (not bit-for-bit, but at least functionally equivalent -- excepting compiler bugs). Why? Because the build happens in an isolated environment where, as far as the build process is concerned (we take advantage of a number of kernel features on Linux (chroot) and OSX (Sandboxes)), the system only consists of the packages explicitly listed as build inputs -- nothing else. Excepting esoteric stuff like someone intentionally trying to subvert chroots and such, if you were to put a gun to my head and tell me that you were going to shoot me if you could find a case of non-determism, I'd shrug. We're serious about that whole determinism thing -- it's a core selling point of Nix, and the reason for Nix's "quirks" (the per package prefixes, hashing of build inputs, chroots, lack of network access, etc).

If the above doesn't make the differences obvious, I'll challenge you to answer some questions (the socratic method):

1. How does Homebrew support having two versions of openssl installed concurrently? Describe the paths involved, how the dynamic linker would be guaranteed to load the respective version of libssl, etc.

2. How does Homebrew ensure packages are built exactly the same way across two different machines (e.g. using the same version of autotools, clang, make, etc)? Be sure to explain how the build is guaranteed to not auto detect/enable stuff because of things present on my system that may not be present on your system. Note that Nix will deterministically fail if you fail to explicitly list all of its dependencies (rather than working because, say, cmake just so happened to be installed on the formula author's machine) -- so you'll need to describe how Homebrew achieves this when running a formula.

Yes, this is what Nix offers. You can trivially have have multiple versions of stuff installed without conflicts. Personally, I have multiple versions of the GHC Haskell compiler, Clang/LLVM, Go, Node, etc. In fact, I keep shell.nix file in each of my repositories at work, and all I have to do is $ nix-shell in each project root and -- like magic -- I have all of the compilers/environment-variables/etc setup for work. Meanwhile, outside of that terminal window, my system is untouched.

Nix is quite different from just about any other package manager out there. If there's something that seems unfeasible or impossible, please feel free to ask specific questions to challenge my assertions. Nix is so different that it will be tempting to think "surely he means... no, that can't be possible, he must be mistaken... but how would that work?" Rather than silently thinking those things, please feel free to skip right to asking "how?"

(For more details, see also my long post to your original statement/question)

> Either we can build everything from source, so that we have reproducibility (see e.g. emerge for a practical system, although it is still not bit-for-bit reproducible) or else we can ship everything as a blob so we have total consistency (e.g. Docker, Go).

Nix is really both of these combined, and is not like apt/dpkg in how it separates source and binary packages.

A Nix package is really instructions (including all compiler flags) of how to build something from source [1] and nothing more. Due to being pure functions, given the binary inputs (e.g. a zlib dependency previously built) and the instructions, any two people will always get the same binary output upon executing those instructions [2].

Furthermore, without executing the instructions, I can generate the unique identifier of the resulting binary package by hashing the inputs and the instructions. This unique identifier can be used to determine if the package already exists locally in /nix/store, and optionally be used to check if some trusted binary cache (such as Nix's binary cache) already has the binary package.

There is no binary package in the sense that apt/dpkg have source packages and many derived binary packages which are installed without need of the source package.

For example (contrived and simplified):

  gcc-uniqueGccHash = bootstrappingProblem()
  libc-uniqueLibcHash = buildLibc(gcc-uniqueGccHash)
  zlib-uniqueZlibHash = buildZlib(gcc..., libc...)
  myprog-uniqueMyProgHash = buildMyProg(gcc..., libc..., zlib...)

To download "myprog" from a binary cache, I can't calculate uniqueMyProgHash using any old versions of libc and zlib; I must have exactly the same hashes that were used by the system that built "myprog", all the way up the dependency chain. And I'll only ever have that if the builds are reproducible and/or I downloaded them all from the same binary cache.

[1] In practice, there's nothing stopping the instructions from saying "download this binary blob" instead of "download this source tarball".

[2] The problem of build reproduciblity is not solved, but many outside of Nix also see this as important, see https://wiki.debian.org/ReproducibleBuilds

It would appear that your primary takeaway is that we put stuff in a different directory, for an apparently arbitrary reason. I agree, that would be silly -- if it were the case.

With /usr/local/{bin,lib,include}, how would you install multiple versions of openssl? Would you violate convention and do something like /usr/local/openssl-x.x.x/{bin,lib,include}? How do you avoid file path conflicts? When a package is built against a particular version of openssl, how do you ensure that version of openssl sticks around, and that that, upon execution of said program, the dynamic linker loads specifically that version of openssl and not one of the others?

While I encourage you to consider those questions, the short answer is "you can't". At least, not with conventional package managers like Homebrew.

Because Nix builds each package in a unique prefix (governed by the hash of all of the build inputs, recursively), the Nix package manager can use chroots (and Sandboxes on OSX) to ensure that packages are deterministic, and that each package refers specifically to the versions of the dependencies they need (dynamic libraries are fixed via rpaths and we apply patches for references to executables -- so "python -m foo" becomes e.g. "/nix/store/-python-2.7/bin/python -m foo").

If you don't use a unique prefix per package, the problem of "how do I install stuff without worrying about path collisions amongst the multiple versions of requisites" becomes intractable. Ditto for build determinism, binary caches, lightweight environments a la nix-shell, etc.

I suppose we could shoehorn each package into /usr/local/--/{lib,bin,include}, and then we could tick the "use an existing directory under /" box, but is that really better? I don't see what we gain from that, since that's already an abuse of /usr/local.

You're very welcome -- thanks for raising the question :)