Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Didn't Need Kubernetes, and You Probably Don't Either

benhouston3d.com

Post reply on HN
Print view View original

341–350 of 436 posts

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#341

Post by p_l »

madjam002 wrote: post #224
noname44 wrote: post #181

Earlier quoted context omitted.

lol, even if you have complex apps there are always easier solutions than Kubernetes. It is evident that you have never run such an app and are just talking about it. Otherwise, you would know the issues you would encounter with every update due to breaking changes. Not to mention that you need a high level of expertise and a dedicated team, which costs far more than running an app on Fargate. Recommending a managed…

I've been running apps on Kubernetes clusters for the past 6 years and the only thing that really comes to mind that was a breaking change was when the ingress class resource type was introduced. Everything else has been incremental. Maybe I'm forgetting something. What's wrong with recommending a managed cluster? I wouldn't use one but it is certainly an option for teams that don't want to spin up a cluster from scr…

There was breaking change around 1.18, which was spread over few releases to make migration easier. Similar fix pattern as with graduating beta to stable APIs for things like Ingress, they just IIRC covered all the core APIs or so? Don't have time to look it up right now.

Generally the only issue was forgetting to update whatever you use to setup the resources, because apiserver auto-updated the formats to the point worst case you could just grab them with kubectl get ... -o yaml/json and trim the read-only fields.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#342

Post by p_l »

torginus wrote: post #214

By the way how much malice can we attribute to cloud vendors? >Moreover, Kubernetes’s slow autoscaling meant I had to over-provision services to ensure availability, paying for unused resources rather than scaling based on demand. A typical Linux instance on AWS starts up in about 8 seconds from the asking to start to command line, so lets double that. You could start up and shut down instances in 15 seconds. Why the…

TL;DR someone didn't learn to plan capacity.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#343

Post by p_l »

oron wrote: post #87

I just use a single k3s install on a single bare metal from Hetzner or OVH, works like charm, very clean deployments, much more stable than docker-compose and 1/10 of the cost of AWS or similar.

Doing the same, grabbed a reasonably cheap Ryzen (zen2) server with 64GB ECC and 4x NVMe SSDs (2x 512G + 2x 1024G).

Runs pretty much this stack:

  "Infrastructure":

  - NixOS with ZFS-on-Linux for as 2 mirrors on the NVMes 
  - k3s (k8s 1.31)
  - openebs-zfs provisioner (2 storage classes, one normal and one optimized for postgres)
  - cnpg (cloud native postgres) operator for handling databases
  - k3s' built-in traefik for ingress
  - tailscale operator for remote access to cluster control plane and traefik dashboard
  - External DNS controler to automate DNS
  - Certmanager to handle LetsEncrypt
  - Grafana cloud stack for monitoring. (metrics, logs, tracing)

  Deployed stuff:
  - Essentially 4 tenants right now
  - 2x Keycloak + Postgres (2 diff. tenants)
  - 2x headscale instances with postgres (2 diff. tenants, connected to keycloak for SSO)
  - 1 Gitea with Postgres and memcached (for 1 tenant)
  - 3 postfix instances providing simple email forwarding to sendgrid (3 diff. tenants)
  - 2x dashy as homepage behind SSO for end users (2 tenants)
  - 1x Zitadel with Postgres (1 tenant, going to migrate keycloaks to it as shared service)
  - Youtrack server (1 tenant)
  - Nextcloud with postgres and redis (1 tenant)
  - tailscale-based proxy to bridge gitea and some machines that have issues getting through broken networks
Plus few random things that are musings on future deployments for now.

The server is barely loaded and I can easily clone services around (in fact a lot of the services above? instantiated from jsonnet templates).

Deploying some stuff was more annoying than doing it by hand from shell (specifically nextcloud) but now I have replicable setup, for example if I decide to move from host to host.

Biggest downtime ever was dealing with not well documented systemd-boot behaviour which caused the server to revert to older configuration and not apply newer ones.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#344

Post by p_l »

Timber-6539 wrote: post #182
doctorpangloss wrote: post #46

Earlier quoted context omitted.

People don’t want solutions, they want holistic experiences. Kubernetes feels bad and a pile of scripts feels good . Proudly declaring how much you buy into feelings feels even better!

Those pile of scripts are the solutions on which Kubernetes was built on. And the fact that they never break is why greybeards still use them over the complexity of shiny new things.

The script break so hard is why one of my first times i felt a "grey beard moment" (with actual grey beard even) was telling a junior the "running ansible from their laptop is not acceptable solution"

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#345

Post by p_l »

nosefrog wrote: post #319
mitjam wrote: post #144

I believe CloudRun is based on KNative which runs on Kubernetes. Thus, you’re still running on Kubernetes, it’s just abstracted away from you.

Nope.

It's not as much mentioned in the docs these days, and Google might have built a more optimized version, but it's still using knative APIs and originally was essentially KNative deployed on GKE just managed by Google.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#346

Post by p_l »

dpeckett wrote: post #237
PohaJalebi wrote: post #124

I like how the author says that Kubernetes has a vendor lock in, yet suggests a GCP managed service as their preferred alternative.

That probably runs on Kubernetes (or Borg) under the hood.

Cloud Run explicitly uses KNative APIs... which are kubernetes objects.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#347

Post by p_l »

wetpaste wrote: post #122

RE: slow autoscaling Maybe the cloud companies could do something here by always keeping a small subset of machines online and ready to join the cluster. Provided there is some compromise in what the configuration is for the end user. I guess it doesn't solve image pulling. Pre-warming nodes is an annoying problem to solve. Best solution I've been able to come up with is: Spegel (lightweight p2p image caching) + Karp…

1. Do some capacity planning

2. Apply appropriate changes to application resources (like parameters for spreading pods around)

3. Add descheduler[1] or similar tool to force redistribution of pods

4. Configure your cluster autoscaling params according to values from step (1) and have it autoscale before nodes are too heavily loaded.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#349

Post by pclmulqdq »

p_l wrote: post #337
pclmulqdq wrote: post #329

Earlier quoted context omitted.

The complexity of k8s comes the moment you need to hold state of some kind. Now instead of one systemd entry, we have to worry about persistent volume claims and other such nonsense. When you are doing things that are completely stateless, it's simpler than systemd.

If you need to care about state with systemd you still have the "nonsense" of persistent volume claims, they are just something you keep in notes somewhere, in my experience usually in heads of the sysadmins or an excel sheet or a text file that tries to track which server has what data connected how.

Understand that in the hypothetical system we are discussing, there are something like 1-2 servers. In that case the "volume claim" is just "it's a file on the obvious filesystem" and does not actually need to be spelled out they way you need to spell it out in k8s. The file path you give in environment variables is where the most up-to-date version of the volume claim is. And that file is free to expand to hundreds of GB without bothering you.

Re: I Didn't Need Kubernetes, and You Probably Don't Either

#350

Post by p_l »

pclmulqdq wrote: post #349
p_l wrote: post #337

Earlier quoted context omitted.

If you need to care about state with systemd you still have the "nonsense" of persistent volume claims, they are just something you keep in notes somewhere, in my experience usually in heads of the sysadmins or an excel sheet or a text file that tries to track which server has what data connected how.

Understand that in the hypothetical system we are discussing, there are something like 1-2 servers. In that case the "volume claim" is just "it's a file on the obvious filesystem" and does not actually need to be spelled out they way you need to spell it out in k8s. The file path you give in environment variables is where the most up-to-date version of the volume claim is. And that file is free to expand to hundreds…

Things get iffier when you start doing things like running multiple instances of something (maybe you're sticking two test environments for your developers), or suddenly you grew a bit or no longer fit on the server and start migrating around.

The complexity of PVCs in my experience isn't really that big compared to this, possibly lower, and I did stuff both ways.

Post reply on HN