Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

github.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#21

Post by solatic »

hinshun wrote: post #19
solatic wrote: post #17

Super interesting. I can clearly see how this improves the caching story compared to vanilla Kubernetes+containerd, and therefore the potential this has to improve autoscaling performance. Of course, it understandably requires the host VM to be NixOS, which means my Kubernetes cluster needs to be running NixOS, which means I would need to abandon the ease afforded by EKS/GKE... which is a tall ask. It would be great…

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service. I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later. In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

I would definitely appreciate seeing some docs with quick pointers on setting that up for EKS at least, if not also GKE!

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#22

Post by adobrawy »

solatic wrote: post #17

Super interesting. I can clearly see how this improves the caching story compared to vanilla Kubernetes+containerd, and therefore the potential this has to improve autoscaling performance. Of course, it understandably requires the host VM to be NixOS, which means my Kubernetes cluster needs to be running NixOS, which means I would need to abandon the ease afforded by EKS/GKE... which is a tall ask. It would be great…

I'm not sure exactly how the communication of components works there, but in k8s can this be deployed using daemonset rather host service?

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#23

Post by mikepurvis »

hinshun wrote: post #19
solatic wrote: post #17

Super interesting. I can clearly see how this improves the caching story compared to vanilla Kubernetes+containerd, and therefore the potential this has to improve autoscaling performance. Of course, it understandably requires the host VM to be NixOS, which means my Kubernetes cluster needs to be running NixOS, which means I would need to abandon the ease afforded by EKS/GKE... which is a tall ask. It would be great…

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service. I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later. In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups.

I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#24

Post by mikepurvis »

yjftsjthsd-h wrote: post #9

So that means that it needs the host to run nix, right? I suppose that could be useful, but if I'm running nix-built images as containers it's usually because I want to run somewhere that's not "nix native".

Even at the cost of the k8s nodes having to be "special", this is still fabulously useful running Nix-native workloads on stuff like GitLab or Jenkins workers.

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#25

Post by rgoulter »

mikepurvis wrote: post #23
hinshun wrote: post #19

Earlier quoted context omitted.

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service. I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later. In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups. I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on e…

> I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time

I got the impression that it uses the node's nix daemon.

From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq

> What's the difference between this and a nix-in-docker?

> If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level.

> nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#26

Post by alex_hirner »

hinshun wrote: post #6
0x457 wrote: post #5

Earlier quoted context omitted.

So if the package isn't in cache, then suddenly k8s node is building it from source?

If you are pushing nix-snapshotter images to a Docker Registry, then it can only either use what's in your host nix store, or fetch from a binary cache. The same is true if you are using the special image reference like `nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar`, it just means that instead of fetching the image manifest from a Registry, you are using the Nix protocols. You will only build…

I think your usage example will be easier to understand, if it explains where nix:0 is setup as CRI image-service. Maybe I glanced over it.

Awesome project!

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#28

Post by throwawayqqq11 »

intelVISA wrote: post #27

Finally, something cool in this space that isn't a naked VC grift. Gl Edgar and Robbie.

I am still sceptical. What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage?

I dont see a killer selling point, just some middle ware with extra containerization features which also could be pulled into nixpkgs.

But i have to admit, in my perception bubble VC gets more desperate and to them, the search metrices for nix must be shining like a christmas tree. I am certainly biased negatively.

Re: Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd

#30

Post by rgoulter »

throwawayqqq11 wrote: post #28
intelVISA wrote: post #27

Finally, something cool in this space that isn't a naked VC grift. Gl Edgar and Robbie.

I am still sceptical. What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage? I dont see a killer selling point, just some middle ware with extra containerization features which also could be pulled into nixpkgs. But i have to admit, in my perception bubble VC gets more desperate and to them, the search metrices for nix must be shin…

> What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage?

This is addressed in the readme FAQ. https://github.com/pdtpartners/nix-snapshotter#faq

If you're running Nix-based images on Kubernetes, seems like this tool has reduced overhead / storage compared to if you build an OCI/Docker image.

Post reply on HN