Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing K3s with vanilla Kubernetes

hoelzel.it

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Comparing K3s with vanilla Kubernetes

#61

Post by b33j0r »

wouldbecouldbe wrote: post #8

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like what... I checked the server. Everything down. I'd been running Kubernetes on Digital Ocean. And Digital Ocean forced a Kubernetes update that was incompatitable with mine at night. Took me 8 hours to fix it. No sleep. Ended up moving it back to a good old VPS. A…

Yep. My problem with a managed cloud is that kubernetes drivers are vendor specific. I hit this landmine too when the ingress I was using got a breaking update (as far as I could tell).

Still worked on my test cluster. It was a throw away site, so I didn’t even bother fixing it. And, I honestly can’t tell you what went wrong. Just deploying docker compose, often with ansible, is the most reliable for me at most scales.

Re: Comparing K3s with vanilla Kubernetes

#62

Post by beebmam »

kobalsky wrote: post #59
davidkuennen wrote: post #41

Earlier quoted context omitted.

Sounds like Digital Ocean was the problem, not Kubernetes. Been running managed Kubernetes in GCP for years without any issues. They are bugging me to update for some time now, but I don't think they would force an update on me.

gke is pretty good at that, if there is a forced upgrade coming they will let you know months in advance.

There isn't a single managed vendor of Kubernetes that won't force an upgrade (or shut down your entire cluster if you don't upgrade). It is too costly for these companies to maintain so many old versions of Kubernetes that are not supported.

If you want a Kubernetes cluster that never gets forcibly updated or shut down, you should not be using a managed service, period.

Re: Comparing K3s with vanilla Kubernetes

#63

Post by ilyt »

b33j0r wrote: post #61
wouldbecouldbe wrote: post #8

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like what... I checked the server. Everything down. I'd been running Kubernetes on Digital Ocean. And Digital Ocean forced a Kubernetes update that was incompatitable with mine at night. Took me 8 hours to fix it. No sleep. Ended up moving it back to a good old VPS. A…

Yep. My problem with a managed cloud is that kubernetes drivers are vendor specific. I hit this landmine too when the ingress I was using got a breaking update (as far as I could tell). Still worked on my test cluster. It was a throw away site, so I didn’t even bother fixing it. And, I honestly can’t tell you what went wrong. Just deploying docker compose, often with ansible, is the most reliable for me at most scale…

Yeah, that's the annoying part. You get around 80% of parts that just work anywhere and rest is entirely dependant on what's underneath k8s or on what cloud it is running.

When going in I assumed there are some kind of standard way to implement ingresses/loadbalancers but no, it's just different plugins each with different syntax and features.

Re: Comparing K3s with vanilla Kubernetes

#64

Post by neuronexmachina »

kobalsky wrote: post #59
davidkuennen wrote: post #41

Earlier quoted context omitted.

Sounds like Digital Ocean was the problem, not Kubernetes. Been running managed Kubernetes in GCP for years without any issues. They are bugging me to update for some time now, but I don't think they would force an update on me.

gke is pretty good at that, if there is a forced upgrade coming they will let you know months in advance.

For anyone curious about the actual schedule: https://cloud.google.com/kubernetes-engine/docs/release-sche...

Re: Comparing K3s with vanilla Kubernetes

#65

Post by amq »

wouldbecouldbe wrote: post #8

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like what... I checked the server. Everything down. I'd been running Kubernetes on Digital Ocean. And Digital Ocean forced a Kubernetes update that was incompatitable with mine at night. Took me 8 hours to fix it. No sleep. Ended up moving it back to a good old VPS. A…

In my experience, helm is the worst offender. Somehow, every second chart update has a breaking change preventing the update, defaults that don't mind wiping the persistent volume and a discontinued somewhat consistent 'stable' central repo make me seriously regret using helm charts for anything that is not ephemeral.

Re: Comparing K3s with vanilla Kubernetes

#66

Post by taspeotis »

bithavoc wrote: post #25
wouldbecouldbe wrote: post #8

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like what... I checked the server. Everything down. I'd been running Kubernetes on Digital Ocean. And Digital Ocean forced a Kubernetes update that was incompatitable with mine at night. Took me 8 hours to fix it. No sleep. Ended up moving it back to a good old VPS. A…

DigitalOcean also cordons Nodes for no reason at random times. Currently moving to EKS exactly for this reason

In late 2021 we had EKS nodes shit the bed for no apparent reason. Max out CPU and couldn't SSH in. They'd get recreated eventually by EC2 (multiple hours) despite being tainted in k8s pretty quickly. We ran with N+2 nodes to deal with it.

After dutifully updating EKS releases and AMIs it seems to have fixed itself last year.

EKS seems to be fine generally, but it would be nice if they didn't release new AMIs with missing commits?? [1] and especially [2]

[1] https://github.com/aws/containers-roadmap/issues/319

[2] https://github.com/awslabs/amazon-eks-ami/issues/278#issueco...

Re: Comparing K3s with vanilla Kubernetes

#67

Post by kobalsky »

beebmam wrote: post #62
kobalsky wrote: post #59

Earlier quoted context omitted.

gke is pretty good at that, if there is a forced upgrade coming they will let you know months in advance.

There isn't a single managed vendor of Kubernetes that won't force an upgrade (or shut down your entire cluster if you don't upgrade). It is too costly for these companies to maintain so many old versions of Kubernetes that are not supported. If you want a Kubernetes cluster that never gets forcibly updated or shut down, you should not be using a managed service, period.

I may have not expressed it well but what I wanted to say was that GKE communicates very clearly when you will be hit with a forced non-patch upgrade, not that upgrades are undesirable. Running a cluster without updates is a deathwish.

Re: Comparing K3s with vanilla Kubernetes

#69

Post by mulmen »

mcmcmc wrote: post #40
mulmen wrote: post #11

Earlier quoted context omitted.

What are the three letters? It only makes sense for being confusing, which is the base state of Kubernetes. "Our name is literally a complex joke about complexity" doesn't inspire confidence in a project that supposedly simplifies complexity.

The three letters are k, 3, and s

The 3 is supposed to be short for three other letters. What are they?
Post reply on HN