Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I think about Kubernetes

garnaudov.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: How I think about Kubernetes

#21

Post by k8ssskhltl »

zsoltkacsandi wrote: post #7

> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach. Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion…

But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s. Or perhaps using non-native tools or wrappers.

Re: How I think about Kubernetes

#22

Post by anymouse123456 »

The allure of declarative approaches to complex problem solving has finally been worn down to nothing for me and Kubernetes was the last straw, nearly 10 years ago.

The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else, invariably generates a horror show of extremely verbose boilerplate, duplication, bloat, delays and pain.

If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.

They'll hum along quietly serving many thousands of actual customers and likely cost less to purchase than a single month (or at worst, quarter) of today's cloud-based muggings.

When you pay, you'll pay for bandwidth and that's real value that also happens to make your work environment more efficient.

Re: How I think about Kubernetes

#23

Post by paddw »

> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster.

This is a pretty good definition.

I think part of the challenge is the evolution of K8s over time sometimes makes it feel less like a coherent runtime and more like a pile of glue amalgamated from several different components all stuck together. That and you will have to be aware of how those abstractions stick together with the abstractions from your cloud provider, etc...

Re: How I think about Kubernetes

#24

Post by DonHopkins »

stavros wrote: post #3
NewJazz wrote: post #2

Love the HN title mod here lol

HN removes a "how" if the post starts with it, sometimes making it hilarious.

Otherwise you might confuse it with the HN mod tomhow.

Same reason they remove "dang" if the post starts with it, like the discussion about "Dang! Who ate the middle out of the daddy longlegs".

https://ifunny.co/picture/dang-who-ate-the-middle-out-of-the...

Re: How I think about Kubernetes

#25

Post by whytevuhuni »

websiteapi wrote: post #4

I always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh. I really like the idea of something like Firebase, but it never seems to work out or just move the complexity t…

> What's the really simple K8s?

I think K8s couples two concepts: the declarative-style cluster management, and infrastructure + container orchestration. Keep CRDs, remove everything else, and implement the business-specific stuff on top of the CRD-only layer.

This would give something like DBus, except cluster-wide, with declarative features. Then, container orchestration would be an application you install on top of that.

Edit: I see a sibling mentioned KCP. I’ve never heard of it before, but I think that’s probably exactly what I’d like.

Re: How I think about Kubernetes

#26

Post by zsoltkacsandi »

szundi wrote: post #13
zsoltkacsandi wrote: post #7

> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach. Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion…

It would have helped if you tell us why you don’t like this approach.

It's right there:

> the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach

But some more concrete stories:

Once, while I was on call, I got paged because a Kubernetes node was running out of disk space. The root cause was the logging pipeline. Normally, debugging a "no space left on device" issue in a logging pipeline is fairly straightforward, if the tools are used as intended. This time, they weren't.

The entire pipeline was managed by a custom-built logging operator, designed to let teams describe logging pipelines declaratively. The problem? The resource definitions alone were around 20,000 lines of YAML. In the middle of the night, I had to reverse-engineer how the operator translated that declarative configuration into an actual pipeline. It took three days and multiple SREs to fully understand and fix the issue. Without such a declarative magic it takes usually 1 hour to solve such an issue.

Another example: external-dns. It's commonly used to manage DNS declaratively in Kubernetes. We had multiple clusters using Route 53 in the same AWS account. Route 53 has a global API request limit per account. When two or more clusters tried to reconcile DNS records at the same time, one would hit the quota. The others would partially fail, drift out of sync, and trigger retries - creating one of the messiest cross-cluster race conditions I've ever dealt with.

And I have plenty more stories like these.

Re: How I think about Kubernetes

#27

Post by zsoltkacsandi »

NewJazz wrote: post #18
zsoltkacsandi wrote: post #7

> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach. Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion…

I feel like the author has a good grasp of the Kubernetes design... What about the approach is problematic? And why don't you think that is how Kubernetes was designed to be used?

I wrote some personal stories below in this thread as a response to another user.

Re: How I think about Kubernetes

#28

Post by zsoltkacsandi »

k8ssskhltl wrote: post #21
zsoltkacsandi wrote: post #7

> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach. Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion…

But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s. Or perhaps using non-native tools or wrappers.

> But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s.

Yes, and 99% of the companies do this. It is quite common to use Terraform/AWS CDK/Pulumi/etc to provision the infrastructure, and ArgoCD/Helm/etc to manage the resources on Kubernetes. There is nothing wrong with it.

Re: How I think about Kubernetes

#29

Post by KronisLV »

websiteapi wrote: post #4

I always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh. I really like the idea of something like Firebase, but it never seems to work out or just move the complexity t…

In ascending order of functionality and how much complexity you need:

  - Docker Compose running on a single server
  - Docker Swarm cluster (typically multiple nodes, can be one)
  - Hashicorp Nomad or K3s or other light Kubernetes distros

Re: How I think about Kubernetes

#30

Post by themgt »

anymouse123456 wrote: post #22

The allure of declarative approaches to complex problem solving has finally been worn down to nothing for me and Kubernetes was the last straw, nearly 10 years ago. The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else, invariably generates a horror show of extremely verbose boilerplate, duplication, bloat, delays and pain. If you're not Google, please for the love of go…

If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.

You can literally get a Linux box (or two) in the corner and run:

  curl -sfL https://get.k3s.io | sh -
  cat
How am I installing a monolith and a database on this Linux box without Kubernetes? Be specific, just show the commands for me to run. Kubernetes that will work for ~anything. HNers spend more tokens complaining about the complexity than it takes to setup.

The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else

Like, brainfuck? Like bash? Like Terraform HCL puppet chef ansible pile-o-scripts? The effort required to output your desired infrastructure's definition as JSON shouldn't really be that gargantuan. You express yourself in anything else but it can't be dumped to JSON?

Post reply on HN