Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google admits Kubernetes container tech is too complex

theregister.com

Post reply on HN
Print view View original

151–160 of 449 posts

Re: Google admits Kubernetes container tech is too complex

#151

Post by matt9ucci »

syrusakbary wrote: post #21

Kubernetes adds a vast amount of complexity, and in my rationale is because it centers scaling on the wrong unit (the Operating System). Docker introduced a great level of abstraction and reproducibility over platforms. However, Docker (or OS-based containers) are the most atomic unit of computation on Kubernetes. Which causes centering scaling on the Instance, instead of the Application or even the functions. This l…

> Which causes centering scaling on the Instance, instead of the Application or even the functions.

That should be why Red Hat took an early investment in Kubernetes. Operating systems did matter for them. Applications did not.

> I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit

I agree with that. The thing I worry about is a k8s variant like Krustlet, running Wasm instead of container, might introduce the same complexity into the Wasm world. We need a more app-centric scaling solution.

Re: Google admits Kubernetes container tech is too complex

#152

Post by regularfry »

dilyevsky wrote: post #88
fen4o wrote: post #13

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store. - I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I nev…

I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...

For two systems of equal functionality, the one that allows or encourages fewer footbullets is the better design. Not all complexity is essential.

Re: Google admits Kubernetes container tech is too complex

#153

Post by diarrhea »

fy20 wrote: post #65
NathanFlurry wrote: post #60

Earlier quoted context omitted.

I’d recommend giving Docker Swarm + Traefik a shot. It’s dead simple to set up manually and has very little “magic” in how things work under the hood. Plus much of your existing Docker Compose config will work out of the box. It vastly simplified the deployment process too. I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed…

I use Swarm at home (only on a single node, because it turns out 3 are overkill for my needs) and it's been running great for 6 months so far. Before that I tried various incarnations of k8s and eventually they'd just destroy themselves up and require a rebuild (the main issue was persistent storage). My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host netwo…

Mind sharing a bit more on your home setup?

Asking because I never saw the point in multiple container replicas for simple self-hosted stuff. One container each has served me well so far [0] (Nextcloud, Bitwarden, GitLab), and if they crash, they just get restarted. Multiple containers increase throughput, supporting more users, is that it? It just sounds nightmarish in regards to storage and parallel, conflicting writes.

[0]: One container per component (web, db, cache, ...).

Re: Google admits Kubernetes container tech is too complex

#154

Post by fendy3002 »

LeonM wrote: post #149

I have a love/hate relation with K8S, I have several customer project that I work on deployed on GKE. For the most part it has been a rock solid set-and-forget experience, but... The problem for me is that these project are not under full time development. Most of them see maybe one or two deployments per year. It seems like every time I do a deployment, some part of the YAML config structure has been deprecated, wit…

> If you are an average Joe developer and you want to use containers in production, just stick with a machine with docker-compose on it. It's a lot easier to maintain and has far fewer surprises down the road.

This is exactly what I am doing. It's easy to configure and almost same configuration between local test and production machines. The only downside is it's hard to monitor whether service is up without additional tools, which I lazy to research.

Additionally if you don't have a good enough ci cd yet you can deploy with volume-linked container.

Re: Google admits Kubernetes container tech is too complex

#155

Post by KronisLV »

sofixa wrote: post #79
NathanFlurry wrote: post #60

Earlier quoted context omitted.

I’d recommend giving Docker Swarm + Traefik a shot. It’s dead simple to set up manually and has very little “magic” in how things work under the hood. Plus much of your existing Docker Compose config will work out of the box. It vastly simplified the deployment process too. I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed…

Docker Swarm is pretty much abandonware/on life support at best, so one should avoid using it for new stuff. Hashicorp's Nomad is the best choice on the complexity for features scale IMHO, and that's why i'm writing an article how great it is, how easier some things are and what's missing compared to Kubernetes.

And yet, Docker Compose is pretty popular for local development, so much so, that it's not uncommon to find a docker-compose.yml in the repositories for many open source projects. And Docker Swarm builds on that, by bridging the gap between Docker Compose and multi-server deployments, with tools like Swarmpit and Podman for easier management of it as well, much like Rancher does for Kubernetes. I agree that Docker Swarm isn't developed as actively as it should be, but disagree that it should be avoided and disagree that it should be allowed to die. In my opinion, it's a more minimalistic and more sane approach to container orchestration with minimal up front investment (just install Docker and edit your Compose files a bit with deploy constraints, you're ready to go).

Hashicorp's Nomad is good if you have a strong engineering department or need to run mixed workloads (e.g. both containers and native processes) because its abstractions are well suited for this, but HCL, their DSL for describing deployments, doesn't map nicely to neither Docker, nor Docker Compose files, knowledgebases or tutorials. Nomad's integration with Consul is a major boon, but the need to run your own CA for safe communication between nodes, Nomad's read-only Web UI, and the oddity of HCL at times also makes it a non starter for me and some other people.

At the end of the day, these are just two data points, sadly the job market for Kubernetes also dwarfs everything else and sadly many companies will be burned by this and will learn nothing at the end of the day. Ideally, i think that the best route would be evaluating the orchestrators and other technologies that you want to use by doing pilot projects and such, and looking at them in real world circumstances, to determine their fit for your goals and needs (Web UIs will matter for some, but not for others, for example; as will onboarding and the need for long term investment vs plug and play).

Edit: as for Kubernetes, personally i find the K3s distribution to be an almost reasonable alternative to Swarm/Nomad, if the situation calls for it: https://k3s.io/

Edit #2: it would actually be pretty awesome to read more about your experience in this article that you're writing!

Re: Google admits Kubernetes container tech is too complex

#157

Post by je42 »

dtech wrote: post #77
trinovantes wrote: post #36

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

Yes, k8s is not built for solo outfits and small projects. In places with 10+ developers and 10+ services on a single cluster it works surprisingly well from the user (developer) side.

I think it is more like: 10+ servers OR 10+ developers. you can have 10+ services as a single developer ;)

Re: Google admits Kubernetes container tech is too complex

#158

Post by machiaweliczny »

endymi0n wrote: post #71
fen4o wrote: post #13

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store. - I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I nev…

I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny). However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. If you actually go all-in…

IMO fixing BEAM VM to easily work on cluster could be better for distributed systems than k8s.

Re: Google admits Kubernetes container tech is too complex

#159

Post by cies »

Cthulhu_ wrote: post #134
cies wrote: post #128

Earlier quoted context omitted.

I once have been told that development teams smaller than 20 developers have no business in using k8s, due to the complexity to brings. If something as essential as the infra so complex it is not readily understood by everyone on the team, a few (more than one) team members need to become the experts on the matter. For small teams this is simply not worth it.

Definitely; if you as a company want to do Kubernetes or even cloud services (beyond the easy managed service like Beanstalk or GCE), you need to have a dedicated expert on it. Or more abstractly, one full time unit (can be distributed). If it's some guy's part time hobby it will not work.

I think this is what went wrong with k8s. I saw lots of interests from hobbyist, and people proposing k8s in small teams. It become often hear that "you do containers in production? use k8s!" That's just a big disappointment waiting to happen.

Re: Google admits Kubernetes container tech is too complex

#160

Post by goatinaboat »

FlyingSnake wrote: post #109

Putting on my Asbestos Longjohns: The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill. After all no one got fired for buying IBM and recommending Kubernetes. Most teams need a simpler deployment strategies that other's have succinctly mentio…

but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill

There are very strong financial incentives for every individual developer and sysadmin to adopt Kubernetes, regardless of the impact it has on the organisation as a whole. In a sense this is engineering reaching the level of corporate maturity of the sales department who will optimise everything for their commission regardless of the organisations ability to deliver it at a profit, or even at all.

Post reply on HN