Is K8s Too Complicated? • Hacker News Board

Re: Is K8s Too Complicated?

#121

Post by AndyNemmity » Sat, May 12, 2018, 8:13 PM UTC

fear91 wrote: post #27

jaequery wrote: post #10

what are some of the downsides of K8? any horror stories?

You need to hire people who are competent with a new, quickly evolving technology. That means it's hard to hire someone good and it's expensive. The upside is that you can replace multiple "normal" ops people with just a few K8 folks but that also requires a bit of culture change of how you do development.

I started a year ago on a team, and there are so many different technologies, things to know, gotchas everywhere, that even competent people can struggle outside of taking a single domain and owning it.

It is not remotely easy to actually use and understand all parts of a massive system, at least ours.

Re: Is K8s Too Complicated?

#122

Post by sandGorgon » Sat, May 12, 2018, 8:15 PM UTC

K8s comes with the inherent dependence that you are using a cloud infra with a lot of the setup done right.

Most importantly - ingress. Getting your ingress on k8s is still a big issue.

I'm not sure about all of these statements that networking in k8s is superior to swarm. For the longest time, it was a huge mess to configure weave vs flannel vs calico. Arguably it was because of the third party implementation themselves, but then I would argue that comparing the superiority of k8s to docker swarm is an apples to oranges comparison...Since swarm's networking was always built in and opinionated.

For a long time , it worked much better than k8s.

Re: Is K8s Too Complicated?

#123

Post by lowbloodsugar » Sat, May 12, 2018, 8:23 PM UTC

pstadler wrote: post #13

hacknat wrote: post #5

Right tool for the right job. Is K8s too complicated? For some use cases it is. They probably should do a better job of discouraging certain use cases, but calling their elevator pitch “bullshit” is hyperbolic. There are exceptions to every rule, but a good rule of thumb is cluster size. If you’re managing less than 25 servers than K8s is probably over kill. As you start to creep north of 40 servers K8s really starts…

Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clust…

You: Look what I did with k8s

Top voted reply: You had to use a thing to do what you want, and that thing isn't even supported any more. [Either I was hallucinating at the time and imagined it, or there was a reply that got deleted]

I think you've just quite nicely demonstrated the original authors point.

Re: Is K8s Too Complicated?

#124

Post by pstadler » Sat, May 12, 2018, 8:26 PM UTC

lowbloodsugar wrote: post #123

pstadler wrote: post #13

Earlier quoted context omitted.

Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clust…

You: Look what I did with k8s Top voted reply: You had to use a thing to do what you want, and that thing isn't even supported any more. [Either I was hallucinating at the time and imagined it, or there was a reply that got deleted] I think you've just quite nicely demonstrated the original authors point.

You sound like you‘re new to this business.

Re: Is K8s Too Complicated?

#125

Post by lowbloodsugar » Sat, May 12, 2018, 8:49 PM UTC

Too complicated for whom?

We just gave up on Mesos. It has been impossible to hire a team to support Mesos, which in many ways, is simpler than k8s while delivering greater scale[1]. But it's been me and one other guy building the whole thing when we planned on having ten. We're planning on replacing it with something else, and New Guy wants that something else to be k8s. Or wanted. Then we spent an hour or so asking "So, how will you handle failure mode (that happened) X?" or "What about the teams that need Y?" and no ECS is looking real good to this guy.

At the literal end of the day, when a bug causes masters to lose quorum at 2am, do you want to be fixing that, or do you want an AWS or Azure team to be fixing that. When you have a hundred apps happily running but the coordination framework is down, so that if you lose and agent, or get lots of traffic, the system that would add or replace capacity is down, how fun is that? Well, we didn't have an outage, because all the apps were up and running, but thank goodness we didn't lose an agent of get a spike in traffic. You live in constant terror, because it never gets so bad (site down), but you have lots of near death experiences where you race against time to fix whatever fun way the system failed. God forbid you upgraded to the release (of Marathon) that has a failure mode of "shut everything down" (we didn't, because I'm "overly cautious", but others did).

Now for some companies, where they have the need for a huge cluster, and the cash and reputation to be able to hire for it, then Mesos or k8s seem like a great idea. But for everyone else, use ECS and have a team at AWS keep that thing running.

[1] I suspect it's probably easier to hire people for k8s, and at the same time more difficult to hire people who can actually do the work. k8s looks simpler than mesos, but is far more complicated. Mesos has a smaller following, but those I've met know what they're doing and respect the problem. And Mesos didn't get everything right, but it's "righter" than k8s IMHO. There's a need for a Mesos 2. =)

Re: Is K8s Too Complicated?

#126

Post by zie » Sat, May 12, 2018, 8:49 PM UTC

justinsaccount wrote: post #117

zie wrote: post #93

Earlier quoted context omitted.

Yes, True enough. Arguably, network policies are outside of Nomad's scope. It's a resource scheduler. Nomad doesn't turn up interfaces for you, or do routing of IP traffic. Nomad is not a one-stop shop, like k8s tries to be, it does resource scheduling in a nice declarative manner, that's about it. It's much more in the unix toolset philosophy, let 1 tool do 1 thing well, and make integration as painless as possible.…

I don't really see how nomad is operationally simpler than k8s. To run a service behind something like traefik on k8s I would: bootstrap a CA for cluster tls run etcd run k8s (apiserver,etc,kubelets) run your application run the traefik ingress controller To run a service behind something like traefik on nomad you would: bootstrap a CA for cluster tls run consul run nomad (servers, clients) run your application run t…

We are talking about different things. I'm talking about keeping Kubernetes/Nomad alive and breathing and happy. The ops part of devops. You are talking about running stuff under them. I agree they are similar in running applications under them.

Operationally simple:

* 1 binary, for both servers and agents. * 1 config file. For consul & nomad 2 config files. * Upgrades are simple, bring down a node, replace binary, start it back up.

Docs are straightforward, it's easy to understand how Nomad works it's not complicated, you can get your head around the server/agent split and nodes and everything in an hour probably (for both nomad AND consul). k8s is a very complex beast, with many, many binaries and configs, there are helpers that get you setup, but they all have their own pros and cons, and sharp edges. Chances are you would not want to use a helper in a production setup, which means you have to understand all those moving parts.

Keeping a k8s cluster running nicely and upgraded consistently requires many full-time admins. Keeping a nomad cluster running requires very little work(I do it part time, maybe an hour a month on a busy month).

Arguably for dev/testing under consul & nomad, you would do this:

  consul agent -dev &
  nomad agent -dev &
  nomad run myapp.nomad
  nomad run traefik.nomad

Adding vault to the mix for secrets: vault server -dev &

For production use it's obviously more involved, but most of that is just around understanding your operational requirements and getting them working with nomad, not really about nomad itself.

Re: Is K8s Too Complicated?

#127

Post by rco8786 » Sat, May 12, 2018, 8:55 PM UTC

matthewmacleod wrote: post #86

rco8786 wrote: post #85

Earlier quoted context omitted.

Maybe. But the question wasn’t about the advantages of k8s over other containerization systems.

I know, you’re right of course. It’s kinda like all those “we rewrote our Ruby app in Go and it’s 1000x faster so Go is awesome” articles. Containerisation is good for those things, but it’s still possible that Kubernetes is an excessively complex solution to the problem!

Fair enough. To your original comment, I was part of a large transition from bare metal servers and custom rolled CI, deployment, monitoring, etc over to using Mesos and the end result was infinitely more pleasurable and productive to work with as an engineer.

Re: Is K8s Too Complicated?

#128

Post by rossdavidh » Sat, May 12, 2018, 9:08 PM UTC

KaiserPro wrote: post #103

K8s is a beast, and it has a fairly specific workflow, which for most people is not a good fit. For example you need to understand that certain workloads dont fit well on the same box (DBs and anything IO/memory sensitive for example) Then there is the "default" network setup where each node is _statically_ assigned a /24. (because macvtap + dhcp is "unreliable" and inelastic apparently.) Now, I've heard a lot of tal…

Wow. I guess I am not the first person to look at K8s and think, "they re-invented the mainframe".

Re: Is K8s Too Complicated?

#129

Post by markbnj » Sat, May 12, 2018, 9:13 PM UTC

AndyNemmity wrote: post #119

markbnj wrote: post #70

Earlier quoted context omitted.

> What you're saying is pretty much the result of my biggest gripe with Kubernetes, though it's one I don't have a lot of ideas of how to fix; there's too much damn boilerplate. 1000 lines of YAML to store maybe 100 relevant lines. I think that's more a helm issue than a k8s issue. I've been using helm in production for over a year and k8s for almost three years. Prior to adopting helm we rolled our own yaml template…

The stable chart thing is so weird. Internally use we some abstractions, but I looks at stable charts and it requires so much time just to understand all of what's going on. Everything is a variable pointed to values, and you can't reason about any of it. It seems like the hope is, just ignore it all, and the docs are good, and just follow them, but I don't live in any kind of world I can do that. And the commits, an…

> It seems like the hope is, just ignore it all, and the docs are good, and just follow them, but I don't live in any kind of world I can do that.

Yep, agreed, we've used very few charts from stable, and in some cases where we have we needed to fork and change them, which is its own special form of suck. The one I contributed was relatively straightforward: a deployment, service and a configMap to parameterize and mount the conf file in the container at start. Even so I found it a challenge to structure the yaml in such a way that the configuration could expose the full flexibility of the binary, and in the end I didn't come anywhere near that goal. You take something like a chart for elasticsearch or redis and its just so much more complicated than that.

Re: Is K8s Too Complicated?

#130

Post by Pneumaticat » Sat, May 12, 2018, 9:17 PM UTC

pstadler wrote: post #13

hacknat wrote: post #5

Right tool for the right job. Is K8s too complicated? For some use cases it is. They probably should do a better job of discouraging certain use cases, but calling their elevator pitch “bullshit” is hyperbolic. There are exceptions to every rule, but a good rule of thumb is cluster size. If you’re managing less than 25 servers than K8s is probably over kill. As you start to creep north of 40 servers K8s really starts…

Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clust…

How did I not find this one month ago, as I set up a two node hobby cluster?!