Live data from Hacker News

Kubernetes is hard

rcwz.pl

131–140 of 164 posts

Re: Kubernetes is hard

#131
post #5

Earlier quoted context omitted.

Yes, they did, but did it really take less effort than running it on Kubernetes?

People think it took less effort. Right up until you needed to do one of the very many things k8s implements. For example, in multiple previous employers, we had cronjobs: you just set up a cronjob on the server, I mean, really, how hard is that to do? And that server was a single point of failure: we can't just spin up a second server running crond, obviously, as then the job runs twice. Something would need to prov…

"Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible"

How are these 'bespoke scripts' but helm charts are not 'bespoke'? Or do you consider them 'bespoke-but-better'?

Re: Kubernetes is hard

#132

I feel with AI, maybe in a couple of years it's going to trivial to deploy things on current infra stacks. AI can probably create whole range of Terraform scripts, deploy k8s and dockers and scale them automatically, with maybe a few humans as supervisor.

I suppose once the AI takes all the ops and dev jobs, we'll just have to seek employment doing the one thing AI can never seem to do - driving a vehicle.

From what I see, dinasauosaurs(gov, schools, hospitals, anything with a union) will continue to be dinaosaurs.

Re: Kubernetes is hard

#133

> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kern…

OMFG so much this. I worked at a large company that deployed it's own Kubernetes stack, on a VERY large number of physical hosts. The theory was that the K8S would simplify our devops story enough that we could iterate quickly and scale linearly. In reality, the K8S team ended up being literally 10x larger than the team building the application we were deploying on it. In addition, K8S introduced entirely new categor…

Using k8s is not remotely the same thing as maintaining your own k8s stack. One is easy (get your feet wet in an afternoon), the other is hard (maybe after a couple months of full-time study you can pull it off).

The vast majority of teams that have enough crap to run to warrant using k8s should not be maintaining their own k8s stack.

In fact, it’s entirely possible that running k8s is so hard that the only players that can do it reliably are the big cloud companies.

Re: Kubernetes is hard

#134

Earlier quoted context omitted.

Cloud services are shit unless I can run them locally when developing and testing.

why on earth do you need a functional implementation of a consumed service for testing?

Iteration speed and blazing fast automated tests. When I discovered minio, I suddenly got much more confident coding against s3.

Re: Kubernetes is hard

#136
post #94
post #86

Earlier quoted context omitted.

You can setup a solid k3s cluster in 30 minutes. I'm sorry you had a hard time but just because you didn't succeed at your attempt doesn't mean it actually is super hard.

Setting it up is relatively easy, keeping it running consistently while navigating the schizophrenic levels of change and indecision on how things are done is another bag of chips. No need for passive aggression.

Yeah we use k3s for local and testing environments, where it shines. I wouldn’t want to use it to run stuff in prod and figure out how to do zero-downtime upgrades with multiple nodes.

Re: Kubernetes is hard

#137
post #78

> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kern…

if you think k8s is the most complex system anyone in the world will ever use: 50K googler using borg beg to disagree.

Actually most devs should have a much easier time with borg than k8s bc there’s an army of sres/devs running it for them. I was one of those running it but even I didn’t appreciate the complexity of running airtight kubernetes distro at the beginning. That’s on onprem tho, if you don’t know how to network just use the cloud thing

Re: Kubernetes is hard

#138
post #78

Earlier quoted context omitted.

if you think k8s is the most complex system anyone in the world will ever use: 50K googler using borg beg to disagree.

This is a bad argument. You are basically admitting that companies need google-level resources to hire dozens (hundreds?) of infra engineers to maintain a K8s-type infra.

Where did I say that k8s needs that? I just said borg is more complex than k8s, so your argument doesn't even hold.

Now, turning things around: companies need to hire more infra engineers, that's true.

Re: Kubernetes is hard

#139

Earlier quoted context omitted.

You're likely looking at the problem from a different angle. Kubernetes is an ecosystem for managing software deployed on a fleet. There's value in that even if it's two or three hosts. What size of an org is too small for Terraform? Puppet? CI/CD? Docker? Some bash scripts? All these tools should fundamentally solve some piece of the puzzle for you. Kubernetes on a managed platform just happens to cover many pieces…

This is the attitude I recoil from. Some of the technologies you listed --- CI is an example --- earn their keep so well that they're essentially universal best practices (at least for serverside software). But others don't, and you can't use a list that includes, like, "source control" as a way to launder in the idea that Kubernetes is a universal best practice. It is not. It earns its keep in some places and not in…

The only best practice I strongly preach is using source control to define your infrastructure all the way down to the versions of binaries serving your customers. Kubernetes is a means to an end there, when used in combination with "gitops". It's very compelling and a very malleable pattern.

As far as I can tell the other alternatives are to either go low and own the kernel on up, which means a lot more depth in your stack. Albeit this is the traditional systems curriculum, it still requires a lot of expertise that is relatively uncommon. I was once like you, I suspect, and our rose-colored glasses for the days of yore make going low more appetizing than I think is reasonable from a business perspective. No one at a startup should ever care about iptables.

The other end of the spectrum is stapling your software to managed services that range from portable (S3) to not (GCP's PubSub). For most startups I'd reckon this is actually the preferred approach as portability can be solved after you find some sort of product market fit. My reservation here is that going this route often blinds folks from understanding how to solve these problems yourself, without the handholding of a big cloud.

Here is where Kubernetes shines. It's the best of both worlds, and the lowest common denominator is relatively portable but at a high enough level that you can be productive. Surely our experiences here differ but mine thus far has been nothing short of positive in the last couple of years. Before then? Mostly a shitshow without deep expertise, but today you can get by without learning much at all.

In comparison to Kubernetes, the closest thing I've used that wasn't kubernetes was a bunch of home-rolled Terraform and puppet and shell scripts. Not too different than what 37signals is bragging about. But it sucked. It's fragile and complex and if the one person who wrote 95% of it leaves your company you are hosed. Kubernetes, when used in an appropriate manner, unshackles you from this sort of thing. Sure, YAML is terrible but at least your YAML looks a lot like my YAML or that YAML on stack overflow.

Kubernetes is fundamentally the next easiest thing behind whatever the equivalent to Heroku/Cloudrun/Fly.io.

If you think it's easier to run a service on a VPS by scp'ing a tarball and ./'ing the server then you're not being honest with yourself about your sysadmin hobby/addiction affecting your work. If you're skilled and competent, good for you! I am too, and we could run out servers happily together forever. But it's sadly a lift to ask newcomers to follow along. The industry is moving away from this level of depth as far as most developers are concerned.

Re: Kubernetes is hard

#140

Earlier quoted context omitted.

You're likely looking at the problem from a different angle. Kubernetes is an ecosystem for managing software deployed on a fleet. There's value in that even if it's two or three hosts. What size of an org is too small for Terraform? Puppet? CI/CD? Docker? Some bash scripts? All these tools should fundamentally solve some piece of the puzzle for you. Kubernetes on a managed platform just happens to cover many pieces…

> Kubernetes is an ecosystem for managing software deployed on a fleet. There's value in that even if it's two or three hosts. there's value yes, but there's also cost, and that cost is huge > Kubernetes on a managed platform just happens to cover many pieces that you still need to solve otherwise i do not need to solve the vast majority of things that kubernetes solves certainly not in 2-3 node deployments, very lik…

It's not a huge cost. This is ludicrous. Sign up for a free account on GCP and walk through their getting started guide and you'll have stuff running in under an hour.

> your "fine" rests on a base level of knowledge which is enormous, certainly incompatible with p99 of users

I doubt it. I have far, far more experience running services on bare metal pizza boxes in 48U racks in PoPs all over the world, but that is a base level of knowledge that actually is enormous and certainly incompatible with p99 of my pool of potential colleagues.

If you can figure out how to run a serverless app, you can figure out Kubernetes. It's not rocket science.

Post reply on HN