Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clusters become personal (like PCs did)

aranya.tech

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Clusters become personal (like PCs did)

#41

Post by stryan »

hedgehog wrote: post #28
stryan wrote: post #25

Earlier quoted context omitted.

As far as I can tell and from some quick researching of the guys previous experience, that's all it is. I think the implication is that LLM's will be architecting and deploying the cluster setups at some point? Which sounds horrific so I'm assuming I am interpreting it long The article itself reminds me of the enthusiasm I felt for plan9 when I first heard about it back in uni. I also thought everyone should have the…

Claude Code + Ansible + whatever stuff needs to be managed gives some visibility and control and in my experience is reliable enough to be useful.

I'm assuming you're at least overseeing the creation/updates of the Ansible playbooks and have some familiarity with what is being managed outside of that. While I personally would not do that[0], I can see the reasoning behind it.

ClusterdOS appears to be a kubernetes-in-a-box multiple node setup that's goal is to work so well that the user doesn't know or care what it's doing. I wouldn't trust an LLM with managing one machine by itself, let alone a whole cluster of them running the incredibly complex mess that Kubernetes is (and that's not even counting the 8 other layers of software this is), so this feels like an order of magnitude worse.

[0] Using LLMs for sysadmin research or boilerplate writing is one thing, but after a certain amount of use you're really just paying $X a month for Anthropic to manage your systems for you. I'd rather just pay a real person to do it at that point. I'd also rather people get over their pathological fear of learning how to run a server but I've given up on that.

Re: Clusters become personal (like PCs did)

#42

Post by 0123456789ABCDE »

skybrian wrote: post #32
Grimburger wrote: post #15

Earlier quoted context omitted.

> Why do I need a cluster? Uptime, self healing, reproducibility, separating the system from app. There's probably a half dozen more. K8s comes with resource consumption tax certainly but for anything beyond the trivial it's usually justified. > Separate VM's for different apps works well for isolation Sounds inefficient along with a lot more work doing the plumbing than simply writing a 100 lines of yaml.

Who wants to deal with YAML? Sometimes the easiest way to set up a VM is by talking to your phone: https://commaok.xyz/ai/just-in-time-software/ I mean, I don't do that, but I'll type a prompt.

you won't have to deal with yaml for these clusters

let me draw this out the way i've been playing with:

a classic vm exists, and supports kvm — this means you can run stuff like firecracker in there

an ssh server runs on this vm, and when you connect to it, you're dropped into a repl/tui where you can list existing microvms, create new ones, or destroy existing ones, and, of particular use, you can attach to one.

as an added nicety, if you connect with `ssh user+dev@example.com`, your connection skips the management interface and you are dropped into the `dev` machine — if it didn't yet exist, you wait 3s, and now it does

vms can talk to each other internally, can connect out, and persist if the server needs a restart

what i don't have yet is proper multi-tenancy, it treats each ssh key as an account, which is fine since it's just me; incoming connections is not figured out, internal supervisor to keep services running inside each microvm, isolation inside firecracker, snapshoting or backups, and the whole shopping list that would make it an actual mvp

Re: Clusters become personal (like PCs did)

#44

Post by hosh »

antonvs wrote: post #36
hosh wrote: post #33

Earlier quoted context omitted.

Not all clusters are elastic. Cloud infrastructure can be, but HPC setups before the cloud were not.

Even in a physical hardware, on-premise scenario, it's still easier to scale horizontally than vertically in almost all cases, for all the reasons I mentioned. That's a big reason why Kubernetes was adopted at an unprecedented pace at medium to large organizations - because it helps manage that approach.

They could have chosen Mesos instead. Kubernetes had other characteristics that allowed it to be adopted far and wide besides the ability to scale horizontally.

Re: Clusters become personal (like PCs did)

#45

Post by wrxd »

plqbfbv wrote: post #40
skybrian wrote: post #2

The article assumes there are people who want clusters. But a single Linux VM in the cloud can scale pretty far. Separate VM's for different apps works well for isolation. Why do I need a cluster?

> Why do I need a cluster? I run a single-node K8s cluster on a dedicated server because it's way cleaner to manage than the previous mess and mix of docker compose + traefik routing + random stuff installed as package on the host. I can create "vhosts" for practically anything in a declarative manner, and if the cluster blows up, I have 5 small scripts to bootstrap it and all I need is `kubectl apply -k .`.

I briefly played with k3s before realising than with a single machine I was maintaining a lot of complexity for limited benefits. Then I switched to NixOS, have everything declared in configuration and a much leaner and simpler setup

Re: Clusters become personal (like PCs did)

#46

Post by druid »

tardedmeme wrote: post #27
DeathArrow wrote: post #10

I don't see how an operating system can work for a cluster. You can have more than one CPU and more than one storage connected to one mainboard and that works because the interconnect fabric is very fast. We don't have have the possibility to connect different computers at the same kind of speed that would let them work together seamlessly.

One could argue that multiple cores are already not seamless especially if you have NUMA (now available in high-end desktops by the way! and every multi-socket system that's ever existed) and the distinction between RAM and disk is very not seamless and so is any other number of things you'd hope the OS would magically handwave away for you but it doesn't. 10Gbps is now very cheap and 100Gbps is viable at hobby scale…

Exactly how I'm thinking about it. NUMA, the RAM/disk hierarchy, CXL. Operators have always abstracted over nonuniform substrates with very different latency tiers. The fabric inside a modern server is already a small network. But the argument for an OS at the cluster level isn't that the interconnect becomes seamless, but that the substrate becomes standardized, regardless of underlying hardware.

Re: Clusters become personal (like PCs did)

#47

Post by skybrian »

0123456789ABCDE wrote: post #42
skybrian wrote: post #32

Earlier quoted context omitted.

Who wants to deal with YAML? Sometimes the easiest way to set up a VM is by talking to your phone: https://commaok.xyz/ai/just-in-time-software/ I mean, I don't do that, but I'll type a prompt.

you won't have to deal with yaml for these clusters let me draw this out the way i've been playing with: a classic vm exists, and supports kvm — this means you can run stuff like firecracker in there an ssh server runs on this vm, and when you connect to it, you're dropped into a repl/tui where you can list existing microvms, create new ones, or destroy existing ones, and, of particular use, you can attach to one. as…

Sounds like a nice setup. The way exe.dev does things seems somewhat similar.

Re: Clusters become personal (like PCs did)

#49

Post by hedgehog »

stryan wrote: post #41
hedgehog wrote: post #28

Earlier quoted context omitted.

Claude Code + Ansible + whatever stuff needs to be managed gives some visibility and control and in my experience is reliable enough to be useful.

I'm assuming you're at least overseeing the creation/updates of the Ansible playbooks and have some familiarity with what is being managed outside of that. While I personally would not do that[0], I can see the reasoning behind it. ClusterdOS appears to be a kubernetes-in-a-box multiple node setup that's goal is to work so well that the user doesn't know or care what it's doing. I wouldn't trust an LLM with managing…

I've been using various UNIXes and clones since the 90s so I do generally know what's going on but I also have no desire to fill my brain with the syntax to the new new new commands to configure an Ethernet interface on Linux etc, or the work necessary to understand fully why VA-API on a certain chip has specific quirks that break freerdp, nor the toil of backporting and patching the necessary libraries, or the specific dance required to set up a machine to TFTP new firmware onto one of the switches, or.... You get the idea.

I'm also not a fan of all the complexity of Kubernetes, one directory with simple to read files makes it a lot more transparent what's where and how it's set up, and the commit history + changelogs make it relatively clear what's changing and why. No distributed database or fancy bootstrapping, just a ubiquitous config format and tool to apply. Changes at the granularity of "a new host is available at A.B.C.D, configure it as a dev server" or "add a new Debian system container named 'blah' to X, bridge it to the research network only, limit to 16 hyperthreads / 64 gb ram, set up for development on git://". It works ok for now.

The next major change will be when models that run locally are capable enough to drive the config changes themselves.

Re: Clusters become personal (like PCs did)

#50

Post by nine_k »

wrxd wrote: post #45
plqbfbv wrote: post #40

Earlier quoted context omitted.

> Why do I need a cluster? I run a single-node K8s cluster on a dedicated server because it's way cleaner to manage than the previous mess and mix of docker compose + traefik routing + random stuff installed as package on the host. I can create "vhosts" for practically anything in a declarative manner, and if the cluster blows up, I have 5 small scripts to bootstrap it and all I need is `kubectl apply -k .`.

I briefly played with k3s before realising than with a single machine I was maintaining a lot of complexity for limited benefits. Then I switched to NixOS, have everything declared in configuration and a much leaner and simpler setup

I think k8s starts making sense when you have to manage more than 10-15 machines. Better yet, 50-100 machines. Especially if these 200 machines actually run 3-4 types of containers total.

Usually it's rather unlike a sane dev setup. Even if your prod setup uses hundreds microservices (you're Google or Uber or something like that), you don't want to run all of them in your personal dev environment, you reuse 90%-99% of stable microservices running in the QA / integration / whatnot environment, and only run a handful locally.

Post reply on HN