Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Is Your Private Cloud

oneuptime.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Kubernetes Is Your Private Cloud

#81

Post by mindcrash »

Speaking as someone who is deploying HA Kubernetes clusters on bare metal just for fun in just a few seconds I noticed something which made me stop reading this article all together.

Just deploy Rook and Ceph? ARE YOU BLEEPING KIDDING ME?!?

There's a job description called "Storage Engineer". These people know a little bit about Kubernetes, but are mostly specialized in everything Ceph. This tells you everything how hard it is to keep Ceph humming along in production. As a sidenote: if you want to make really good money there's also somebody called a "Ceph consultant" who is called in when SHTF. And if SHTF in a Ceph cluster, it really does.

And that's besides all the crap it takes to get and keep Kubernetes running smoothly: Kernel Optimization. Networking. Security. Storage integration. Observability. And the list goes on...

In other words, unless you are VERY well versed in a variety of topics ranging from server architecture to deep Linux knowledge and are knee deep in the usual day to day operations stuff already you are better off running Kubernetes in the cloud and leaving all the intricacies to the likes of Google, Microsoft and Amazon than trying to run a well designed cluster architecture yourself. It just isn't worth it.

Re: Kubernetes Is Your Private Cloud

#82

Post by rozhok »

proxysna wrote: post #61

Been using Nomad and different sorts of K8S for the last 6+ years at home/work. Nomad is easier to bootstrap, lighter on resources and so so so much easier wrap your head around. Just Nomad + NFS server, from my perspective, is a perfect start for a homelab/small project. You can add complexity to it as you go. It is a real joy work with once you after a day of tinkering. Want to run on windows? Sure. BSD, there is a…

Have you managed to setup CI/CD with Nomad? Last time i've checked it was non-trivial task.

Re: Kubernetes Is Your Private Cloud

#83

Post by nostrebored »

almosthere wrote: post #7
nostrebored wrote: post #4

“Everything You Expect from a Cloud, Running on Your Terms“* Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is

It's crazy to me how many people deploy unmaintainable spaghetti mess in all other environments I've been in. "koober" environments are the most organized.

I find this to be an insane argument. I have never seen a k8s environment that wasn’t a garbage heap. But everyone is eager to tell you what a nice can they’ve found next to the rotting banana.

Onboarding new team members? A disaster. The design? All done by one dumpster diver with nobody to call them out on the mistakes because they have no idea what the hell is happening.

I’ve never seen a k8s shop where there weren’t a few principal engineers being roped from incident to incident because teams couldn’t manage their own.

With cloud providers, even ones we don’t use, people have a general idea of how the primitives work. With any reasonably skilled team you get back to talking about how your product works instead of talking about persistent volume replication for Postgres backups or the differing file system behavior or what cli everyone should be using to manage your entire company’s software or how to stop someone from deleting the entire of everything you own or or or

Re: Kubernetes Is Your Private Cloud

#84

Post by pat2man »

bakies wrote: post #57
pat2man wrote: post #42

As someone who self hosted bare metal Kubernetes on my own rack, it's a lot of work to get it set up. We used RedHat Openshift which has a pretty good solution out of the box, but the learning curve was relatively high. That being said, once it was set up, there was not a lot of maintenance. Kubernetes is quire resilient when set up properly, and the cost savings were significant.

Try talos next time. It took minutes to setup. Red Hat docs and product names scare me since they are intentionally obtuse. I thought I wanted openshift but no way i'm paying and i couldn't figure out how to even get started. Talos was such a breeze.

The thing with OpenShift (https://github.com/okd-project/okd) is you can set it up, and then run basically one command (oc new-app .) to push almost any app to Kubernetes. All bells and whistles included.

Re: Kubernetes Is Your Private Cloud

#85

Post by nostrebored »

dilyevsky wrote: post #30
nostrebored wrote: post #4

“Everything You Expect from a Cloud, Running on Your Terms“* Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is

It’s a well known thing that if you run on ec2 they handle all those things for you (especially the security part)

They do manage almost all of them for you. If you’d like we can get some of the EC2 engineers on our team to tell you how.

If you don’t think IAM is a better set of semantics for securing your infrastructure than nightmarish k8s rbac i really don’t know what to tell you.

Can you think back to your batch and honestly tell me that the companies that used k8s are in a better place?

Re: Kubernetes Is Your Private Cloud

#86

Post by proxysna »

rozhok wrote: post #82
proxysna wrote: post #61

Been using Nomad and different sorts of K8S for the last 6+ years at home/work. Nomad is easier to bootstrap, lighter on resources and so so so much easier wrap your head around. Just Nomad + NFS server, from my perspective, is a perfect start for a homelab/small project. You can add complexity to it as you go. It is a real joy work with once you after a day of tinkering. Want to run on windows? Sure. BSD, there is a…

Have you managed to setup CI/CD with Nomad? Last time i've checked it was non-trivial task.

At work i use terraform to configure nomad/consul combo, so basically just a tf with Gitlab.

At home i am using this approach. Dumb, but works well. https://royportas.com/posts/simple-gitops-with-nomad

Re: Kubernetes Is Your Private Cloud

#87

Post by ofrzeta »

ForHackernews wrote: post #77
ofrzeta wrote: post #72

Earlier quoted context omitted.

People also avoid it because it is a fringe system and there's much more knowledge and tools around Kubernetes.

You say that like it's a good thing https://www.netlify.com/v3/img/blog/cncf-landscape-map-2020....

Fair enough, that's also a way at looking at this.

Re: Kubernetes Is Your Private Cloud

#88

Post by spwa4 »

bakies wrote: post #75
eikenberry wrote: post #73

Earlier quoted context omitted.

But 99% of the time you don't outgrow it and don't have SLA's requiring them to have failover or HA. This is why so many sites can get away with using PostgreSQL with it's complete lack of good/native HA.

HA and failover, to me-in these small deploys, is more about hardware maintenance rather than maintaining SLAs. Being able to shutdown the computer hosting your containers to scale vertically.

Single-site-2-absurdly-intense-days failover, yes. With weeks of cleanup afterwards.

What every company seems to want: multisite, multimaster immediate failover, no.

Also kubernetes buys you scaling. Compute. Disk. Database (with help). Etc.

Now I rail against companies for wanting that, and I think you're right. Your webshop does not need that. It has so many moving parts the redundancy will cause more outages than it solves. But you can do this, and so people will pay for it.

It is a technical accomplishment.

And with sufficiently good sysadmins, it can work well, and scale.

Re: Kubernetes Is Your Private Cloud

#89

Post by jameson »

nostrebored wrote: post #4

“Everything You Expect from a Cloud, Running on Your Terms“* Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is

Ops cost to number of server to manage is logarithmic but cloud cost is linear, so there's an intersection it starts to make less sense for cloud. Also equipment depreciation is tax deductible whereas cloud bill isn't. A year of EC2 instance bill is comparable to buying the equivalent server

Also there are vendors renting out datacenters so you do less of hardware management.

Having worked at two companies spending 250M+ on cloud bills alone, they try hard to decouple from cloud but many things are vendor locked

Hybrid has been the answer to both. It shouldn't be a binary decision. stateless compute workload can fairly easily be offloaded to private cloud.

Re: Kubernetes Is Your Private Cloud

#90

Post by rozhok »

proxysna wrote: post #86
rozhok wrote: post #82

Earlier quoted context omitted.

Have you managed to setup CI/CD with Nomad? Last time i've checked it was non-trivial task.

At work i use terraform to configure nomad/consul combo, so basically just a tf with Gitlab. At home i am using this approach. Dumb, but works well. https://royportas.com/posts/simple-gitops-with-nomad

Thank you! That's pretty clever, will try it.
Post reply on HN