Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Nix Based Environments in Kubernetes

flox.dev

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Run Nix Based Environments in Kubernetes

#41

Post by jeremy_flox »

jeremy_flox wrote: post #32
nixosbestos wrote: post #28

Earlier quoted context omitted.

> leverages the unique way that Flox environments are rendered without performing a nix evaluation" I'm curious! and ignorant! help! Is that via (centrally?) cached eval? or what? there's only so much room for magic in this arena.

Yeah, it's essentially cached eval, the key being where/how that eval is stored. When you create a Flox environment, we evaluate the Nix expressions once and store the concrete result (ie exact store paths) in FloxHub. The k8s node just fetches that pre-rendered manifest and bind-mounts the packages with no evaluation step at pod startup. It's like the difference between giving the node a recipe to interpret vs. givi…

Correction: we don't eval when you create environments.

Our catalog continuously pre-evaluates nixpkgs in the background. 'flox install' just selects from pre-evaluated packages -- no eval needed, ever. The k8s node fetches the manifest and mounts the packages.

Eval is done once, centrally, continuously. So... even more pre-val'd, so to speak.

Re: Run Nix Based Environments in Kubernetes

#42

Post by jeremy_flox »

nickysielicki wrote: post #31

What constraints/coordination exists with this, in terms of host driver support? What enforces that Nix does not attempt to use a newer cuda toolkit on a host with an older cuda driver?

You pin the CUDA toolkit version compatible with your driver; manifest.lock ensures zero drift.

Driver version is host-managed (stable), toolkit is hash-pinned (stable). No drift on either side.

Initial selection matters (pick compatible versions, nothing automatic will stop you from initially installing the wrong thing), but once pinned, and unless/until you change it, you get the same toolkit forever regardless of catalog updates or time passing.

Re: Run Nix Based Environments in Kubernetes

#43

Post by d3Xt3r »

__MatrixMan__ wrote: post #39
d3Xt3r wrote: post #33

Earlier quoted context omitted.

> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. One my issues with Nix is the black box that is the store, and maybe it's just my system, but ove…

Have you found that nix-store --gc breaks things, or are you concerned that it's not an aggressive enough garbage collection? When I have space problems, I run that and they're gone, then later I do it again. It could be that I'm just avoiding functionality that it breaks though.

There's still leftovers. For me, it's not about space problem, but more of wanting to run a tight ship. Most recently, I tried running all the stuff on the wiki[1] but I still had leftovers that wouldn't go away no matter what (these were some CUDA dependencies, if it makes a difference). In the end I ended up blowing away my entire Nix install, manually deleting the store and reinstalling Nix - which isn't exactly ideal - considering that the whole point of Nix is to be deterministic and reproducible and all that jazz... so to me it doesn't make sense that it dirties the host and doesn't clean up after itself. Since then I've gone back to using containers as at least they don't pollute the host, and I feel like I'm in greater control over the entire environment.

[1] https://nixos.wiki/wiki/Cleaning_the_nix_store

Re: Run Nix Based Environments in Kubernetes

#44

Post by antonvs »

eep_social wrote: post #26
antonvs wrote: post #24

Earlier quoted context omitted.

Kubernetes can be a godsend at larger orgs. We have six dev teams and are just about done with migrating to k8s. It's an immense improvement over what we had before. It's a version of Greenspun's tenth rule: "Any sufficiently complicated distributed system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Kubernetes."

I think six dev teams is small in terms of kube. I wouldn’t be surprised if that’s close to the perfect size to move onto kube and create and adopt a standard set of platform idioms. at orgs significantly larger than that, the kube team has to aggressively spin out platform functions that enable further layering or risk getting overwhelmed trying to support and configure kube features to cover diverse team needs (tha…

> I think six dev teams is small in terms of kube.

I don't doubt it. By "larger" I just meant larger than something like "running my servers on FreeBSD/OpenBSD and jails or VMM respectively" above, which sounds like a one-person operation.

> I wouldn’t be surprised if that’s close to the perfect size to move onto kube and create and adopt a standard set of platform idioms.

My previous position was at a company about 5x the size, with many loosely related enterprise and government products that they sold into markets in at least 20 countries. They also used k8s quite effectively.

But, I think the key is that you mentioned "the kube team". Having a single team responsible for everything k8s-related at a large org is likely to make it difficult to be effective.

For supporting individual dev teams I think you need people on those teams who have at least some of the necessary knowledge, so they're not entirely dependent on a central team. Even a watered-down version of what devops was supposed to be about is better than nothing.

Re: Run Nix Based Environments in Kubernetes

#45

Post by jeremy_flox »

robinhoodexe wrote: post #34
jeremy_flox wrote: post #29

Jeremy from Flox, here, I want to chime in here so Ron can be with his family, even though he will no doubt be right back on here: Re: Relationship to nix-snapshotter and prior art This is original work, though very much built on prior innovations. Our approach hooks into the upstream containerd runc shim to pull the FloxHub-managed environment and bind-mount the closure at startup. The key distinction is that we use…

First, congrats on the release. I’ve looked at flox and devenv for nixifying our container builds. Our distribution of languages is about 40/30/20/10 of Python, F#, R and nodejs. A dilemma I’m facing is that the win from nix in terms of faster builds and smaller images would be largely from python and R images (where the average size is often 1Gi or larger). However, the developers that use Python or R are less likel…

We don't currently scrape R, it's on the list, for sure.

Re: Run Nix Based Environments in Kubernetes

#46

Post by neobrain »

d3Xt3r wrote: post #43
__MatrixMan__ wrote: post #39

Earlier quoted context omitted.

Have you found that nix-store --gc breaks things, or are you concerned that it's not an aggressive enough garbage collection? When I have space problems, I run that and they're gone, then later I do it again. It could be that I'm just avoiding functionality that it breaks though.

There's still leftovers. For me, it's not about space problem, but more of wanting to run a tight ship. Most recently, I tried running all the stuff on the wiki[1] but I still had leftovers that wouldn't go away no matter what (these were some CUDA dependencies, if it makes a difference). In the end I ended up blowing away my entire Nix install, manually deleting the store and reinstalling Nix - which isn't exactly i…

I feel your pain on this one despite being fairly comfortable with nix by now. This is 100% an issue either the documentation or the nix CLI should do a better job at.

The wiki rightfully points towards "roots", i.e. references produced by nix-build or similar. Additionally, there are other places that will keep references and hence block garbage collection though:

1. Your nix profile (`nix profile list` / `nix profile remove`) and its old generations (`nix profile (wipe-)history`)

2. Your NixOS configuration (configuration.nix) and its old generations (`nixos-rebuild list-generations`)

It doesn't help that there's no discoverable way to tell why a particular nix store path is not being garbage collected either.

Re: Run Nix Based Environments in Kubernetes

#47

Post by __MatrixMan__ »

d3Xt3r wrote: post #43
__MatrixMan__ wrote: post #39

Earlier quoted context omitted.

Have you found that nix-store --gc breaks things, or are you concerned that it's not an aggressive enough garbage collection? When I have space problems, I run that and they're gone, then later I do it again. It could be that I'm just avoiding functionality that it breaks though.

There's still leftovers. For me, it's not about space problem, but more of wanting to run a tight ship. Most recently, I tried running all the stuff on the wiki[1] but I still had leftovers that wouldn't go away no matter what (these were some CUDA dependencies, if it makes a difference). In the end I ended up blowing away my entire Nix install, manually deleting the store and reinstalling Nix - which isn't exactly i…

I salute your desire to run a tight ship. If there were a leak in the nix store, I'd never know until I my garbage collection stopped solving the problem and I'm sure it would be at the worst possible time. If such a leak exists and is found, it'll be by someone running a tight ship.

Instead of running a tight ship I spend a lot of time dreaming about alternate computational universes, and one I particularly like is where new hard disks come pre-loaded with a fragment of bits deemed culturally valuable. Wikipedia and a well curated subset of nixpkgs would be a fine start to such an archive. In this world your files don't grow/shrink to consume/yield empty space, but rather the boundary between your data and that drive's shard of the public archive shifts in one direction or another. This way you or somebody in your neighborhood is likely to already have the file you need, so you can get it from them instead of the internet. Better for being able to roll with the punches if the internet is partitioned.

I don't worry about the size of the nix store because according to this weird fantasy of mine it's on the side of my disk that shrinks when I add files to it; it's not the contained object, but the gas that expands to fill the rest of the container. Not by accident, but as part of a redundant worldwide distributed cache that we put together after deciding that servers controlled by people we don't know are not to be relied on.

Re: Run Nix Based Environments in Kubernetes

#48

Post by the_real_cher »

CuriouslyC wrote: post #7
the_real_cher wrote: post #5

[flagged]

You can say what you want about Kube (it's a bit of a necessary evil for the people that need it), but keep Nix's name out yo damn mouth. It's for real.

Its the React.js of Linux distros complete with the same type of fans.

Re: Run Nix Based Environments in Kubernetes

#49

Post by nixosbestos »

jeremy_flox wrote: post #30
nixosbestos wrote: post #27

So, nix-snapshotter? Also, Flox going all in on "environments" seems like such a choice. I'm sure that Flox is not encouraging shipping a binary-in-a-devshell to Prod, so it seems an interesting branding decision. It's hard for me to understand if I should be excited about this. I think companies do themselves such huge disservices from not being transparent to the nerds that WILL be the ones helping choose/implement…

Totally valid - we buried the lede here. Quick version: Not nix-snapshotter because we skip Nix eval entirely and get way better cache sharing across unrelated workloads (quantized catalog means everything shares base deps). On "environments": these aren't devshells-as-prod, they're the actual runtime; same as 'flox activate' works everywhere. You're shipping a declarative, hash-pinned runtime that happens to also wo…

Hard to say because I'm a bit ignorant of the edges of Flox right now! This comment is great though. :) And even more exciting to hear it will be OSS'd. Appreciate your replies here, good luck!
Post reply on HN