Live data from Hacker News

Nixery – Docker images on the fly with Nix

nixery.dev

61–70 of 90 posts

Re: Nixery – Docker images on the fly with Nix

#61
post #49

I like how they optimize for layer reuse, but at the same time, nix doesn't really fit docker layer caching because it's better. With a unique prefix per package, you don't need stacking of layers. You can just download a bunch of packages in parallel, extract them in parallel, and finally "merge" the prefixes to get those combined bin/ and lib/ dirs.

> You can just download a bunch of packages in parallel, extract them in parallel, and finally "merge" the prefixes to get those combined bin/ and lib/ dirs. docker pull achieves the same result: layers are fetched in parallel, and they are extracted using pgiz (parallel gzip). It just uses a pre-defined order, which does not harm performances, but it is not useful either in case nixery is used.

The point is not about parallelization, it's that nixery has to optimize for cache reuse, which is an artificial problem created by docker.

If you have two layers installing an individual packages like /nix/store/x and /nix/store/y, stacking them as [x, y] and [y, x] would result in the same docker image contents, but docker will generate different hashes.

Re: Nixery – Docker images on the fly with Nix

#62
post #55

Earlier quoted context omitted.

Okay, sorry to hijack, but I keep trying it, and keep getting stuck. Most recently, how do I install a Rust binary from Github? They have a releases page, or I can just do a cargo build. Either way, I would just drop the resulting binary in /usr/bin and it's done. With Nix... I'm totally stumped. Do I have to package it myself somehow?

If you just want to take a precompiled binary and install it, you just have to write a derivation wrapper around the binary which will declare the expected hash and take care of e.g. unzipping and moving to $out/bin. Here is a more involved example of downloading a release and then extracting the binary from a .pkg file: https://gist.github.com/J-Swift/364a8b158bf0b603f6e784e454ca... Here is a more simplified example…

Once I discovered `lib.fakeSha256`, I just put that in the derivation, try to build it and then use the error message to find the correct hash to put into it. Probably not the fastest way to do it, but it's easier for me to remember.

Re: Nixery – Docker images on the fly with Nix

#63

Earlier quoted context omitted.

If you just want to take a precompiled binary and install it, you just have to write a derivation wrapper around the binary which will declare the expected hash and take care of e.g. unzipping and moving to $out/bin. Here is a more involved example of downloading a release and then extracting the binary from a .pkg file: https://gist.github.com/J-Swift/364a8b158bf0b603f6e784e454ca... Here is a more simplified example…

Once I discovered `lib.fakeSha256`, I just put that in the derivation, try to build it and then use the error message to find the correct hash to put into it. Probably not the fastest way to do it, but it's easier for me to remember.

yeah that seems way easier, thanks for the tip!

Re: Nixery – Docker images on the fly with Nix

#64
post #19

Earlier quoted context omitted.

Do you actually use nix and have an experience to share?

Not OP, but the evangelism about "never having build problems anymore" does perplex me a bit. In the languages I have been programming in (Haskell, Ruby, some Python and elm and JS and Rust), I can't recall having any significant build problems in the last ~8 years or so anyway. What does everyone do that their build keeps breaking?

One reason you might have avoided problems in Haskell is that it (specifically: Cabal) added "Nix-style builds" in 2016 ;)

https://cabal.readthedocs.io/en/latest/nix-local-build-overv...

Re: Nixery – Docker images on the fly with Nix

#65

I don’t know if it’s just because I hopped on the bandwagon in the past few months, but it really is starting to feel like Nix is gaining momentum. You can use it on Mac, WSL, Ubuntu multiuser, Docker, in an nspawn container, or through NixOS on a drive. You can build a livecd iso or a Raspberry Pi image from the same flake that you use for everything else. I work in embedded with Yocto every day, and I can’t help bu…

I was afraid of Nix before I adopted it based on what I've read. Now that I've taken the leap, there is no going back. Other operating systems are crude dinosaurs in comparison. Once you get past the learning curve and the initial setup (which can be steep), your system will be far more stable and easy to maintain than anything out there. Declarative OS builds are the future, whether it's Nix or something else.

I actually hate Nix.. and i agree, i'm never going back. I use it on my desktop (Linux), my Macbooks, and i want it on my Windows machine (not that it's going to happen hah).

The thing that Nix, for me, currently fails at is introspection. Every function is a black box and i have no clue what's in it. I have to go dig up files on github to see what it even accepts. It's as if everything is obtuse.

A "simple" LSP/Type system would do wonders for understanding what the function is, what it does, and the inputs/outputs.

.. also i have some difficulty understanding functional "mutation" patterns like how overloads are implemented. But i hope that'll make sense eventually.

NOTE: i also think Flakes are absolutely necessary for Nix. Nix is way less valuable to me without Flakes.

Re: Nixery – Docker images on the fly with Nix

#66
post #54

Earlier quoted context omitted.

Do you actually use nix and have an experience to share?

Yeah I was in a company using Nix for around 18 months where some of the DevOps team were contributors so everything was Nix’d. CI, dev environments, deployments, cluster management. Want to add a library to your Python app? Don’t use pip or poetry, update Nix. But because DRY this nix isn’t even in your project, it’s in another repo somewhere. Want to update Haskell? Well you can’t use Cabal or Stack, you need to us…

> when the total Nix lines of code > 10k for a project then have fun!

I have the suspicion that you’re talking about auto-generated Nix code. For comparison, lock files for language-specific package managers can easily exceed 10k lines. But I have yet to see any hand-written Nix build description for a single software project reach nearly as much LOC.

Re: Nixery – Docker images on the fly with Nix

#67
post #4

Oh great! More indirection. Now when I want to deploy my web app I can check my private nixery.dev deployment is properly configured in Nix to build my Docker images so I can deploy my containers to the cloud so someone can access me Rest API. And the cost of guaranteeing builds will probably work? Running your own nixery service, learning Nix, and learning Docker. I would love someone to do a cost-benefit analysis o…

> I would love someone to do a cost-benefit analysis of these sorts of tools against the time of using (and sometimes debugging) Make and/or Bash. Nix does essentially the same job as Make. The differences are: - Make embeds a shell code interpreter, whilst Nix just execs a binary; given its path, a list of args and a set of env vars. (Note that almost all Nix definitions use bash as their binary!) - Make does meta-p…

Thank you for this post. This is the kind of authoritative, insightful, contextually relevant information that makes HN so valuable.

Re: Nixery – Docker images on the fly with Nix

#68

Earlier quoted context omitted.

I was afraid of Nix before I adopted it based on what I've read. Now that I've taken the leap, there is no going back. Other operating systems are crude dinosaurs in comparison. Once you get past the learning curve and the initial setup (which can be steep), your system will be far more stable and easy to maintain than anything out there. Declarative OS builds are the future, whether it's Nix or something else.

I actually hate Nix.. and i agree, i'm never going back. I use it on my desktop (Linux), my Macbooks, and i want it on my Windows machine (not that it's going to happen hah). The thing that Nix, for me, currently fails at is introspection. Every function is a black box and i have no clue what's in it. I have to go dig up files on github to see what it even accepts. It's as if everything is obtuse. A "simple" LSP/Type…

Guess I know what I’m learning this weekend.

Anyone have tutorials they want to share?

Re: Nixery – Docker images on the fly with Nix

#69
post #44

Earlier quoted context omitted.

> and the initial installation experience on Mac has been all over the place quality-wise in recent years. In minor defense of this, Apple making the root read-only threw a pretty big wrench in things. There are a lot of little reasons (some understandable and some exasperating) it took so long to adjust, but all of them were exacerbated by the amount of new logistical complexity required (and the amount of experimen…

Most of the issues come from the lack of enthusiasm for moving the Nix store somewhere else than /nix . Even with read-only root, macOS has some designated locations where you can write. E.g. Homebrew uses /opt/homebrew , which is fine because /opt is writable. I understand the reasoning of avoiding this on Intel Macs, since there there are years of cached derivations which would become useless without hacks. However…

I guess this is a valid way to frame the problem (and I personally agreed with moving it), but I'd also quibble a bit...

- IIRC, ~stakeholders weren't keen on relocating it just on macOS because it would require a separate set of build/cache infrastructure (and it sounded like the macOS+Nix community would be on the hook for supporting it).

- There did actually seem to be a fair amount of support for moving it globally (and Eelco, while skeptical, didn't sound like he was going to stand in the way), but the coordination work sounded significant to me.

- Also, I think the circular arguments around relocating /nix played their own role in the inaction/bystanding that let the problem fester (as did, to be fair, fear/uncertainty about whether Apple would later ~secure whatever new location was chosen).

For some general references on the above, see

https://github.com/NixOS/nix/issues/2925#issuecomment-499517...

https://github.com/NixOS/nix/issues/2925#issuecomment-523340...

https://github.com/NixOS/nix/issues/2925#issuecomment-549184...

https://github.com/NixOS/nix/issues/2925#issuecomment-549858...

https://github.com/NixOS/nix/issues/2925#issuecomment-550106...

https://github.com/NixOS/nix/issues/2925#issuecomment-550211...

https://github.com/NixOS/nix/issues/2925#issuecomment-625855...

https://github.com/NixOS/nixpkgs/issues/95903#issuecomment-7...

Re: Nixery – Docker images on the fly with Nix

#70
post #54

Earlier quoted context omitted.

Do you actually use nix and have an experience to share?

Yeah I was in a company using Nix for around 18 months where some of the DevOps team were contributors so everything was Nix’d. CI, dev environments, deployments, cluster management. Want to add a library to your Python app? Don’t use pip or poetry, update Nix. But because DRY this nix isn’t even in your project, it’s in another repo somewhere. Want to update Haskell? Well you can’t use Cabal or Stack, you need to us…

At my company, I use nix to manage "external dependencies" but still just use language tools for language-specific dependencies. So for instance, I pull Ruby, plus all the common C libraries used in Ruby native extensions into a shell.nix, but otherwise the Ruby workflow is identical to any other.

Same with JVM stuff. We use Bazel for that. But I use nix to install bazel + a java toolchain.

Nix works so much better than Homebrew, since its easy to pin to an exact commit of nixpkgs.

I definitely wouldn't want to force Nix into well established language workflows, but I am extremely pleased with it for managing package dependencies and development environments in a reproducible way. I'd love to extend it to building Docker images, but I haven't made it that far yet. :)

Post reply on HN