Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will Nix Overtake Docker?

blog.replit.com

Post reply on HN
Print view View original

211–220 of 259 posts

Re: Will Nix Overtake Docker?

#211

Post by KronisLV »

mayli wrote: post #90
cbrewster wrote: post #26

Earlier quoted context omitted.

Author here. As with most things, its all about the trade-offs. Docker has certainly proved itself and that approach has worked on a massive scale. However, its not a silver bullet. For us at Replit, our Docker approach was causing issues: our base image was large and unmaintainable and we had almost no way of knowing what changed between subsequent builds of the base image. We've been able to utilize Nix to address…

Correct, that's one of the cases where docker's layered image system doesn't work well. Nix is almost the perfect tool to perform incremental builds and deployments for the Replit requirements. I wish that docker has the ability to merge multiple parent layers like git, then you can build the gigantic image by just updating single layer. The only hack the docker can do is multistage-build, however that won't work rel…

Disclaimer: the following is still experimental, and will probably remain so for a while.

There is actually the --squash command that you can use during builds, to compress all of the layers: https://docs.docker.com/engine/reference/commandline/build/#...

For example:

  $ docker build --squash -t my-image .
In practice it can lead to smaller images, though in my experience, as long as you leverage the existing systems in place efficiently, you end up shuffling around less data.

E.g.:

  - layer N: whatever the base image needs
  - layer N+1: whatever system packages your container needs
  - layer N+2: whatever dependencies your application needs
  - layer N+3: your application, after it has been built
That way, i recently got a 300 MB Java app delivery down to about a few dozen MB actually being transferred, since nothing in the dependencies or the base image needed to be changed since, it just sent the latest application version, which was stored in the last layer.

Also, the above order also helps immensely with Docker build caching. No changes in your pom.xml or whatever file you use for keeping track of dependencies? The cached layers on your CI server can be used, no need to install everything again. No additional packages need to be installed? Cache. That way, you can just rebuild the application and push the new layer to your registry of choice, keeping all of the others present.

Using that sort of instruction ordering makes for faster builds, less network traffic and ergo, faster redeploys.

I even scheduled weekly base image builds and daily builds to have the dependencies ready (though that can largely be done away with by using something like Nexus as a proxy/mirror/cache for the actual dependencies too). It's pretty good.

Edit: actually, i think that i'm reading the parent comment wrong, maybe they just want to update a layer in the middle? I'm not sure. That would be nice too, to be honest, though.

Re: Will Nix Overtake Docker?

#212

Post by amarshall »

amarshall wrote: post #107
qudat wrote: post #87

As far as I understand nix, the nix examples do not create a reproducible environment because it is not locking the versions down for each package, including nixpkgs. https://nix.dev/tutorials/towards-reproducibility-pinning-ni...

Ass that article says, it’s possible but awkward. It gets better with Flakes https://nixos.wiki/wiki/Flakes

*As (oops, mobile typing)

Re: Will Nix Overtake Docker?

#213

Post by 112233 »

rgoulter wrote: post #115
beermonster wrote: post #110

Earlier quoted context omitted.

Is it? Or is it just a tool that requires a different mindset so those coming to it from another regular distro? If the latter, then I’m not sure it’s something they will see as a problem but more its very ethos.

Maybe an apt comparison is git. It's just as easy to make a mistake with git today as it was however many years ago; git hasn't fundamentally changed in ways that make it easier. Git still more/less requires you to have a good understanding of what's going on in order to be comfortable using it. But, since use of git is now widespread, it's less of an issue. And the git CLI has seen some UX improvements. Nix is very…

> It's just as easy to make a mistake with git today as it was however many years ago; git hasn't fundamentally changed in ways that make it easier.

Easier? Who needs that? Soon then you will have common unlearned folk and peasants trying to use git.

They had the super obtuse git reset, that was four different things bolted together, so they fixed it by adding git restore, that does slightly different things, but you still need both...

Re: Will Nix Overtake Docker?

#214

Post by lolpython »

soraminazuki wrote: post #186
lolpython wrote: post #99

Earlier quoted context omitted.

> I don't understand how these are comparable They fulfill similar business functions - allowing you to run the same code on a bunch of dev machines and on prod (modulo modifications for e.g. database storage in Docker's case). Nix people get hung up on the fact that Docker runs containers, but it doesn't really matter that much. Often Docker is the shortest path to getting software running on multiple machines repro…

> and if there are conflicts between dependencies of the services they can go into different containers. Getting the benefit of Nix - reproducibility - without the extra effort To be clear, are you suggesting that RUN sudo apt-get update && sudo apt-get -y install ... is somehow reproducible? I'm asking because I was surprised to see the above as being described as "reproducible" of all things. Splitting that into ma…

I am frustrated that you ignored my main argument about Docker sufficiently serving the same business function as Nix, with lower maintenance. Instead you focused on a semantic argument about one word in my post - "reproducible". You have the wrong idea about reproducibility, where you say basically everything that is not Nix or Guix is not reproducible. This ignores things like conda and techniques like package version pinning that allow researchers and businesspeople to get the same results from the same code. Here's a definition of "reproducible" from Wikipedia:

Any results should be documented by making all data and code available in such a way that the computations can be executed again with identical results.

https://en.wikipedia.org/wiki/Reproducibility

-----

> To be clear, are you suggesting that > RUN sudo apt-get update && sudo apt-get -y install ...

No, you need to pin the dependency versions. With Python this practice is already normalized with requirements.txt or conda yml files. So you would take an existing project and do:

    RUN conda env create -f environment.yml
which would likely be copy-pasted from the project README. The yml file specifies version numbers for dependencies. The SAT solver is deterministic. For other languages like C maybe the project didn't specify dependency version. So you need to figure them out when you first get a successful build, then specify their versions in apt. You can specify version numbers in the apt-get install line.

Yes, this is reproducible. Definitely good enough for most business use cases. When I say reproducible I do not mean ivory tower math proof reproducible. I just mean that the code will run on the relevant machines they are targeting. As I wrote in my initial comment. And as I defined at the top of this comment.

Also Nix provides a worse experience for pinning dependency versions since it does not have a native concept of version numbers [0]. Instead people have to grep through the Nixpkgs repo to find the correct hash of their dependency version.

> This is even more true for Nix, which has the largest and most up-to-date package repositories out there

No, Docker has the closure (to borrow Nix's terminology) of all of the package managers in that graph. If you add the height of the Debian packages with Pypi and Hackage you already have Nix beat. You can keep adding - cargo, ruby gems, etc all in their native package managers. If Nix were better off then people would be adapting Nix packages to other ecosystems. But the reality is the other way around.

> Plus, with Nix, you can easily make a new package based on existing packages with a mere few lines of code if the existing packages doesn't fit your needs. Other package managers besides Guix doesn't offer you that flexibility so you'd have to compile from scratch

With Nix, you are forced to make new packages based on existing packages. That is not a benefit. Regarding "if the existing packages doesn't fit your needs", compiling from source is not a big deal since Docker caches output artifacts.

[0]: https://github.com/NixOS/nixpkgs/issues/93327

Re: Will Nix Overtake Docker?

#215

Post by clem »

carlmr wrote: post #193
pjmlp wrote: post #165

Earlier quoted context omitted.

I avoid most git usability issues by using it as SVN. When something goes wrong I just bork the whole repo and clone it again, then manually merge the last set of saved changes.

Since learning about reset --hard, I need about 80% less of that.

"git reflog" has also been a blessing in getting out of scrapes.

Re: Will Nix Overtake Docker?

#216

Post by cbrewster »

treis wrote: post #198
cbrewster wrote: post #66

Earlier quoted context omitted.

What happens when you need to update some dependency within that image? Now you have to do an image rebuild. If you're lucky only the top-most layers will be rebuilt and the base layers stay the same, if you're unlucky nearly the whole image is rebuilt. Usually we just want to update some subset of dependencies, but during the rebuild some other dependencies may get updated unintentionally (eg if they aren't pinned t…

That's not really what a reproducible build is though. Reproducible builds are you get the exact same thing from your build script today or three weeks for now. Getting unexpected changes with an updated dependency is a different problem than not having a reproducible build.

Fair, but its still a real issue and solved in a similar way: Nix has finer grained reproducibility -- not only at the environment level but also at the derivation level. Being able to pick and choose which dependencies to update while ensuring other packages are left exactly the same is valuable to us.

Re: Will Nix Overtake Docker?

#217

Post by verdverm »

throwaway894345 wrote: post #197
Nullabillity wrote: post #181

Earlier quoted context omitted.

You can use Nix as a better docker build, see https://grahamc.com/blog/nix-and-layered-docker-images or https://nixery.dev/ .

The dream is using something like Nix to not only reproducibly build a container image but also all of the infrastructure manifests which reference it. I _think_ this is achievable in Nix if you're willing to deal with all of the pain of actually using Nix; however, this would depend on pushing an image to a container registry as part of the Nix build and I'm pretty sure that violates Nix's idioms/conventions? I've c…

See https://dagger.io, the next gen tool by the creators of docker that does this.

CUE + BuildKit

Re: Will Nix Overtake Docker?

#218

Post by oauea »

taeric wrote: post #185
oauea wrote: post #180

Earlier quoted context omitted.

If you can do it in minutes on your machine, you can spend those minutes updating your Dockerfile to automate the steps instead. It's essentially the same thing.

Unless auth is concerned. There are loads of tools I run that I want to just be me. Not whatever user is configured in the image. And learning how to manage that mapping was a heck of a time sink.

How you going to run those tools on CI? Or by your colleagues? These are all questions that need answering anyway, regardless of your usage of Docker.

Re: Will Nix Overtake Docker?

#219

Post by oauea »

kryptk wrote: post #184
oauea wrote: post #182

Earlier quoted context omitted.

Just build your own images like you would otherwise install the software on bare metal. Base those images on official images, not community images.

YMMV. Personally, you would have to pry my Bitnami images out of my cold dead hands.. there is just no way my team of 2 can do anywhere near as good.

Fair, there are a few high quality "vendors" that we also trust, and Bitnami is one of them. RandomJoe42/SomeImage is out, though.

Re: Will Nix Overtake Docker?

#220

Post by greatgib »

soraminazuki wrote: post #200
greatgib wrote: post #117

Along my life, i have worked with a lot of build systems, but I find Nix syntax and commands to be completely awful! I see that it could be useful, but it completely turns me down immediately.

The Nix syntax being complex is something that is taken for a given in online circles, but it's actually one of the most simple languages that I've learned. The Nix syntax is in essence just JSON with functions which consists of the following building blocks: * Primitive values (strings, numbers, paths, booleans, null) * Lists and sets * Variables * Functions * Conditionals * Assertions * Arithmetic operators And tha…

It is not necessary what it can do the problem, but how it does it.

Looking at the page you link for example: https://nixos.org/manual/nix/stable/expressions/expression-s...

>

>

Strange, but let's say ok. But then, in the example, where does the function end?

Then, >

Looking at that, I say whaaaattt? "derivation" does not mean anything obvious to anyone used to build packages. And then, as the subtitle said, did they decide to use a weird name just to confuse everyone for the pleasure?

Then: >

So, looking at the code, from far, function, sets, everything looks mixup ...

Post reply on HN