Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Is Your Private Cloud

oneuptime.com

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Kubernetes Is Your Private Cloud

#61

Post by proxysna »

Been using Nomad and different sorts of K8S for the last 6+ years at home/work. Nomad is easier to bootstrap, lighter on resources and so so so much easier wrap your head around. Just Nomad + NFS server, from my perspective, is a perfect start for a homelab/small project. You can add complexity to it as you go. It is a real joy work with once you after a day of tinkering. Want to run on windows? Sure. BSD, there is a driver for it. Don't want Docker? There is Podman driver. OCI sucks? Just run binaries without isolation. Need VM's? You can switch from Proxmox to purely Nomad setup with Qemu driver with a bit of sweat. Illumos zones on OmniOS? weird, but with quite a bit of time, but there was a repo on github, just need to build the binary with the patch.

And while k8s can do all the same things and much more with a bit of trying, but it requires a mission control the second you add a second developer, you will have built-in primitives that will compete all the time with the ones you bolt-on, etc etc. Nomad feels much more opinionated and in a good way.

Nomad is one of those things that gets you 90% of the way with 20% of effort, and only then if you need something, you can add things to it. K8S is great, way more flexible, there are managed options out there, massive ecosystem, but it always feels like out of the box you need to glue 5 different tools to it, just get it going.

Also Incus. Stephane Graeber is doing lords work by sticking to his thing. That's also super fun to mess with.

Re: Kubernetes Is Your Private Cloud

#62

Post by ecshafer »

senderista wrote: post #55
ecshafer wrote: post #36

Earlier quoted context omitted.

Peak Utilization is a tough one for on prem and is a decent argument for cloud. I was at a company that was also at <10% peak utilization most of the time. It was finance, so it was mostly doing nothing, except for the couple days a year where we shot up 10000x, so we had to build for that case. So yeah losing the data centers, and cloud was a cost savings.

So the "obvious" (but complex!) solution is a "hybrid cloud": use on-prem for the predictable, constant "DC component" of your demand, and use cloud for cyclical or unpredictable demand changes. That will keep peak demand decoupled from permanently provisioned capacity while saving on always-utilized capacity. Easier said than done, of course.

The route we ended up going, was hybrid cloud with a colocation for on-prem (Mainframes and a few servers), and then AWS for the cloud portion. Not sure on what the cost savings were. As I wrote the authentication service I knew what those numbers were, so I knew that on a normal day we would be sitting at 1% cpu usage basically all day except market open/close. And then fiscal quarter ends were a big bump. And then of course big news days. 99% of the time for 99% of the days were just nothing.

Re: Kubernetes Is Your Private Cloud

#63

Post by senorrib »

senderista wrote: post #55
ecshafer wrote: post #36

Earlier quoted context omitted.

Peak Utilization is a tough one for on prem and is a decent argument for cloud. I was at a company that was also at <10% peak utilization most of the time. It was finance, so it was mostly doing nothing, except for the couple days a year where we shot up 10000x, so we had to build for that case. So yeah losing the data centers, and cloud was a cost savings.

So the "obvious" (but complex!) solution is a "hybrid cloud": use on-prem for the predictable, constant "DC component" of your demand, and use cloud for cyclical or unpredictable demand changes. That will keep peak demand decoupled from permanently provisioned capacity while saving on always-utilized capacity. Easier said than done, of course.

K8s helps reduce that complexity a lot.

Re: Kubernetes Is Your Private Cloud

#64

Post by huntaub »

throwawaypath wrote: post #13

Managed Ceph in the past. I cannot comprehend someone putting up with the headache that is Ceph in their home lab. To each their own!

The root cause here is just that managing any kind of storage service is instantly painful. The property of "not losing data" means that you are sort of required to always be doing something in order to keep it healthy.

Re: Kubernetes Is Your Private Cloud

#66

Post by bakies »

devsda wrote: post #47
bakies wrote: post #31

Earlier quoted context omitted.

I'm not really clear on the complaint. Is it immutability or not? I'm not saying delete the cluster and start over, I'm saying i can yank a node or destroy a container without (much of) a consequence. Talos is immutable similarly to nix afaik

I guess the complaint is that with resources being immutable, the only standard & recommended way to deal with a problem is to take the resource out. I know that is the whole point of sheep vs pets but it somehow became the "did you restart the pc" version for operations.

There's only small parts of the typically used parts of the kubernetes api that are immutable and those have good reasons. So I'm still not really sure what issue you're describing.

Re: Kubernetes Is Your Private Cloud

#67

Post by bakies »

stackskipton wrote: post #56
bakies wrote: post #27

Earlier quoted context omitted.

> we are at sub 10% peak utilization of compute resources so... you bought way too much hardware?

That's not unusual. First off, sometimes 1000 extra dollars will get you a ton more compute you need so why not and second, on prem tends to be extremely inelastic so you buy a ton of compute because you never know when compute requirements change.

If we're talking on the scale of $1,000s then it's cheaper to run on-prem than in the cloud. It's really easy to spend $1,000 on managed kubernetes and have very little actual compute.

Re: Kubernetes Is Your Private Cloud

#68

Post by avtar »

nimbius wrote: post #10

Kubernetes is powerful, yes. it is also a feckless rats nest of bolt-ons and ride-alongs. its sharepoint levels of byzantine tuning so complex that, like sharepoint, it comes with its own bespoke administrators that often have little or no knowledge of basic networking or operating systems --only kubernetes--. - Upgrading a kubernetes cluster may as well be an olympic sport. its so draconian most best practice docume…

> what your company likely needs is some implementation of libvirtd or proxmox. run your workloads on rootless podman or (god forbid) deploy to a single VM.

Even with a single VM, someone's company probably will also want a reverse proxy and certificate management (assuming web services), automated deployments, provide secrets to services, storage volumes, health checks with auto restarts, ability to wire logs and metrics to some type of monitoring system, etc. All of this is possible with scripts and config management tools but now complexity is being managed in different ways. Alternatively use K3s and Flux to end up with a solution that checks all of those boxes while also having the option to use k8s manifests in public clouds.

Re: Kubernetes Is Your Private Cloud

#70

Post by pyrale »

bushbaba wrote: post #22
pyrale wrote: post #9

Earlier quoted context omitted.

All things people used to own 10 years ago. It’s not like the people doing that stuff have vanished. Cloud’s big promise was speed to market and price, and let’s be honest, price is no longer there compared to a decent operation. The one thing where clouds remain kings is speed for small teams. Any large enough company should probably Ask themselves whether running their own operation using ias would be a better choi…

My company is on prem, spending north of 1 billion per year. Cloud is actually cheaper when considering total cost of ownership. Thats salaries, opex, capex costs. Worse, our speed to delivery is generally worse. Because on prem is inelastic, we are at sub 10% peak utilization of compute resources. If we added in the likely higher cloud utilization rate we are talking of 30%+ savings from on prem.

My comparison was cloud vs renting metal, not buying your own machines.
Post reply on HN