What constraints/coordination exists with this, in terms of host driver support? What enforces that Nix does not attempt to use a newer cuda toolkit on a host with an older cuda driver?
Run Nix Based Environments in Kubernetes
31–40 of 50 posts
Re: Run Nix Based Environments in Kubernetes
#32Earlier quoted context omitted.
Wrong. If you know nix then you know "leverages the unique way that Flox environments are rendered without performing a nix evaluation" is a very significant statement.
> leverages the unique way that Flox environments are rendered without performing a nix evaluation" I'm curious! and ignorant! help! Is that via (centrally?) cached eval? or what? there's only so much room for magic in this arena.
Yeah, it's essentially cached eval, the key being where/how that eval is stored.
When you create a Flox environment, we evaluate the Nix expressions once and store the concrete result (ie exact store paths) in FloxHub. The k8s node just fetches that pre-rendered manifest and bind-mounts the packages with no evaluation step at pod startup.
It's like the difference between giving the node a recipe to interpret vs. giving it a shopping list of exact items. Faster, safer, and the node doesn't need to know how to cook (evaluate Nix). I don't know, there's a metaphor here somewhere, I'll find it.
Only so much room for magic, for sure, but tons of room for efficiency and optimization.
Re: Run Nix Based Environments in Kubernetes
#33When I worked on an enterprise data analytics platform, a big problem was docker image growth. People were using different python versions, different cuda versions, all kinds of libraries. With Cuda being over a gigabyte, this all explodes. The solution is to decompose the docker images and make sure that every layer is hash equivalent. So if people update their Cuda version, it result in a change within the Python l…
I was an early and enthusiastic adopter of docker. I really liked how it would let me use layers to keep track of dependency between files. After spending a few years using nix, the docker image situation looks pretty bonkers. If two files end up in separate layers, the system assumes dependency so if the lower file changes you need to build a separate copy of the higher one just in case there's actual dependency the…
> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme.
One my issues with Nix is the black box that is the store, and maybe it's just my system, but over time I find it full of redundant files / orphans and no obvious way to flatten it or clean it safely without breaking something.
I wonder how flox solves this.
Re: Run Nix Based Environments in Kubernetes
#34Jeremy from Flox, here, I want to chime in here so Ron can be with his family, even though he will no doubt be right back on here: Re: Relationship to nix-snapshotter and prior art This is original work, though very much built on prior innovations. Our approach hooks into the upstream containerd runc shim to pull the FloxHub-managed environment and bind-mount the closure at startup. The key distinction is that we use…
First, congrats on the release. I’ve looked at flox and devenv for nixifying our container builds. Our distribution of languages is about 40/30/20/10 of Python, F#, R and nodejs.
A dilemma I’m facing is that the win from nix in terms of faster builds and smaller images would be largely from python and R images (where the average size is often 1Gi or larger). However, the developers that use Python or R are less likely to “get” the point of Nix and might have a steeper learning curve than F# developers (where the builds are quite efficient).
That was the context, my question is, how’s the integration with Flox and R/RStudio? I know there’s Rix[1] for managing R packages with Nix.
Re: Run Nix Based Environments in Kubernetes
#35Earlier quoted context omitted.
I was an early and enthusiastic adopter of docker. I really liked how it would let me use layers to keep track of dependency between files. After spending a few years using nix, the docker image situation looks pretty bonkers. If two files end up in separate layers, the system assumes dependency so if the lower file changes you need to build a separate copy of the higher one just in case there's actual dependency the…
> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. One my issues with Nix is the black box that is the store, and maybe it's just my system, but ove…
Both fair points. The README rebuild issue is a Nix hiccup we don't solve; our quantized catalog reduces cascading rebuilds from upstream churn, but input over-specification is still there.
On store bloat: Flox makes it clearer what's in use (explicit environments vs. implicit dependencies), but you still need nix-collect-garbage.
The store accumulates cruft, that's Nix reality, we haven't changed it.
Re: Run Nix Based Environments in Kubernetes
#36I used to love both, Kubernetes and Nix. But after a few years of using both I felt like the abstraction levels are a bit too deep. Sure, it's easy to stand up a mail server in NixOS, or to just use docker/kubernetes to deploy stuff. But after a few years it felt like I don't have a single understanding of the stack. When shit hits the fan, it makes it very difficult to troubleshoot. I am now back on running my serve…
Kubernetes can be a godsend at larger orgs. We have six dev teams and are just about done with migrating to k8s. It's an immense improvement over what we had before. It's a version of Greenspun's tenth rule: "Any sufficiently complicated distributed system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Kubernetes."
No, most large orgs do not need it.
Re: Run Nix Based Environments in Kubernetes
#37Earlier quoted context omitted.
Kubernetes can be a godsend at larger orgs. We have six dev teams and are just about done with migrating to k8s. It's an immense improvement over what we had before. It's a version of Greenspun's tenth rule: "Any sufficiently complicated distributed system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Kubernetes."
No, most large orgs do not need it.
Completely unsupported assertions are the least interesting kinds of comment anyone can post.
Besides, this particular comment would need to explain why it’s likely that its unsupported opinion is correct, rather than all the counterexamples that exist in the actual highly competitive industry that’s heavily using this system.
There’s a fundamental conflict there that doesn’t work in favor of your inchoate opinion. The most likely conclusion is that there are factors at work here that you don’t understand.
Re: Run Nix Based Environments in Kubernetes
#38When I worked on an enterprise data analytics platform, a big problem was docker image growth. People were using different python versions, different cuda versions, all kinds of libraries. With Cuda being over a gigabyte, this all explodes. The solution is to decompose the docker images and make sure that every layer is hash equivalent. So if people update their Cuda version, it result in a change within the Python l…
Yes, there were various attempts to do this in the container ecosystem, but there is a hard limit on layers on Docker images (because there are hard limits on overlay mounts; you don't really need to overlay all the Nix store mounts of course as they have different paths but the code is for teh geenral case). So then there were various ways of bundling sets of packages into layers, but just managing it directly throu…
And I'm back in the land of the living. Can't really beat a response from Justin Cormack!
Re: Run Nix Based Environments in Kubernetes
#39Earlier quoted context omitted.
I was an early and enthusiastic adopter of docker. I really liked how it would let me use layers to keep track of dependency between files. After spending a few years using nix, the docker image situation looks pretty bonkers. If two files end up in separate layers, the system assumes dependency so if the lower file changes you need to build a separate copy of the higher one just in case there's actual dependency the…
> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. One my issues with Nix is the black box that is the store, and maybe it's just my system, but ove…
Have you found that nix-store --gc breaks things, or are you concerned that it's not an aggressive enough garbage collection?
When I have space problems, I run that and they're gone, then later I do it again. It could be that I'm just avoiding functionality that it breaks though.
Re: Run Nix Based Environments in Kubernetes
#40Earlier quoted context omitted.
> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. One my issues with Nix is the black box that is the store, and maybe it's just my system, but ove…
Both fair points. The README rebuild issue is a Nix hiccup we don't solve; our quantized catalog reduces cascading rebuilds from upstream churn, but input over-specification is still there. On store bloat: Flox makes it clearer what's in use (explicit environments vs. implicit dependencies), but you still need nix-collect-garbage. The store accumulates cruft, that's Nix reality, we haven't changed it.
Just to follow up on this, Flox puts packages in one group by default so they share dependencies, plus our quantized catalog means way less version spread than raw Nix. So I do think we still improve on the Nix story, here.
We're also adding "stabilities" (downsample from daily to weekly/monthly snapshots) to reduce churn even more. Still need GC, but a lot fewer bags on trash day. .