Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google admits Kubernetes container tech is too complex

theregister.com

Post reply on HN
Print view View original

341–350 of 449 posts

Re: Google admits Kubernetes container tech is too complex

#341

Post by FlyingSnake »

tylermenezes wrote: post #325

As a small nonprofit with a tech team consisting of mostly volunteers, a few interns, and a small % of our paid staff time, this was a big concern for us. Things are always fine when you're starting, but if you don't understand the system it's hard to troubleshoot when it breaks. Understanding Kubernetes is hard. Obviously there are ways to outsource cluster management, but we don't have the budget of a VC funded sta…

As someone helping a nonprofit with their engineering needs, I’m also looking into Nomad+consul. Do you have any pointers for me?

Re: Google admits Kubernetes container tech is too complex

#342

Post by justin_oaks »

zug_zug wrote: post #334
anchochilis wrote: post #311

Earlier quoted context omitted.

I don't know what to tell you except that with years of experience implementing this infrastructure outside Kubernetes, letting Kubernetes handle it is cheaper. That doesn't mean Kubernetes is a good fit for every organization or workload. You can roll your own anything with enough time and manpower. Whether it makes sense to do so depends on your circumstances.

Well for example, I'm a lead devops on a team of 2-3. We have 6 core services. For the last several years, due to static load we've been fine with 2 instances per service. Now that we want autoscale (even though all the real load is really just the DB), it seems we could get ourselves onto AWS autoscale in ~1 month, though it would require some coding. Spending 3 devops on 1 year doing something is a red-flag to me.…

I agree with your point overall, I just want to add that some DevOps efforts aren't to save DevOps time, but to prevent errors.

When you make a change manually, there's a chance that you forget something, have a typo, etc. Those problems disappear when those same things are automated.

Re: Google admits Kubernetes container tech is too complex

#343

Post by specialp »

reader_1000 wrote: post #327
specialp wrote: post #278

Earlier quoted context omitted.

Yes because whatever you used for app specific configuration like libraries and packages is now done in the Dockerfile and containerized. So the same thing run locally is run in the cloud. Then as far as the infrastructure for running code such as load balancers, service discovery, docker.. That is all given to you just by running K8s. So you are more concerned with shipping immutable containers to k8s than provision…

> So the same thing run locally is run in the cloud Who is preparing Dockerfiles? Developers and system administrators / security people do not generally prioritize same things. We do not use k8s for now (therefore I know very little about it), so this might not be relevant but how do you prevent shipping insecure containers?

Generally developers. When running in a container most of the attack surface is the app itself, and if it is compromised the damage is supposed to be limited to the container. There have been container escape exploits in the past though. But with a container you treat the container as the thing that you run and give resources to and don't trust it just like if you were running an application. All of the principles of giving an application resources such as least privilege apply to containers too.

But since you are not running multiple things or users in one space in a container, something such as an out of date vulnerable library can't be leveraged to gain root access to an entire host running other sensitive things too.

In Kubernetes and docker in general one container should not be able to compromise another, or k8s. But there are other issues if an attacker can access a running container such as now having network access to other services like databases. But again these are all things that can be locked down and should be even if provisioning hosts running things.

Re: Google admits Kubernetes container tech is too complex

#345

Post by jayd16 »

bob1029 wrote: post #176

I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…

Even if you run one container, containers let you use off the shelf apps instead of only off the shelf libs.

I'm not interested in being "not lazy." I only care about user value and ability to provide user value (tech debt/cost/velocity).

For the price of an M5A.16xlarge to get those cores, I can get like 27 m4.larges and have way more fault tolerance.

I think your disdain is misplaced.

Re: Google admits Kubernetes container tech is too complex

#346

Post by specialp »

Ianvdl wrote: post #294
specialp wrote: post #278

Earlier quoted context omitted.

Yes because whatever you used for app specific configuration like libraries and packages is now done in the Dockerfile and containerized. So the same thing run locally is run in the cloud. Then as far as the infrastructure for running code such as load balancers, service discovery, docker.. That is all given to you just by running K8s. So you are more concerned with shipping immutable containers to k8s than provision…

Thanks for the detailed response. It looks like I've still got a lot to learn - I've just lately been playing with LXC to get more familiarised with containers. I've previously looked at Helm apps and they seemed to be very similar to Puppet manifests. From what you said it seems like the approach is to have immutable containers for each application, set up via Dockerfiles, which somehow also simplifies the upgrade p…

You get the load balancing for free in K8s and rolling deploys. What you do is upgrade the deployment with a new docker image, and yes immutably it is replaced. In a case of an HTTP service, k8s will wait until a pod (container) responds healthy until it is put in the loop. Then it steps down old pods according to your rolling deploy metrics and replaces them. You can define what that is like having a max number of pods with a minimum number of pods up.

Re: Google admits Kubernetes container tech is too complex

#347

Post by ogre_codes »

sa46 wrote: post #86
ogre_codes wrote: post #5

I think the single biggest mistake people make with Kubernetes is implementing it too soon. Last company I worked for spent piles of time fighting K8s when a simple well implemented cluster would have done the job. It makes a lot of sense to build portable infrastructure, but you can scale a long ways with much simpler technologies.

I hear this refrain consistently on HN but of the options I've tried, GKE is the most pleasant for my single person app. - App Engine has a bunch of weird limitations and slow deploy times. Qualitatively, it feels like the spotlight has moved on. - Running my own compute instances felt like reinventing Kubernetes, especially once you roll your own deploy mechanism and throw load balancing in the mix. I also don't buy…

You are mistaking my point here. I'm not suggesting you replace K8s with some other thing which does the same things k8s does, but that you build out a simpler/ easier to maintain solution until you need something like k8s. Many Many Many businesses can do just fine without the kind of complexity K8s (or App Engine or GKE) brings to the table.

There are a lot of businesses that will never have to deal with dynamic scaling, engineering autoscaling into those solutions is pointless. If you need that kind of scaling then K8r is fantastic. My point is a lot of people turn to K8s well before they need to or without understanding why they might need it.

Re: Google admits Kubernetes container tech is too complex

#348

Post by throwaway894345 »

vp8989 wrote: post #250
z9e wrote: post #233

Earlier quoted context omitted.

... Until they hire you back because entropy exists :P I'm on the DevOps side as well going through the same transition, k8s also allows insane customization, and I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating. The k8s scene seems to be filled with constant scope creep and refactoring to get it just…

"I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating." My impression is that the primary purpose of Kubernetes is to give SRE teams political air cover to rewrite a lot of their existing processes. Whether Kubernetes is actually required for that, or even net superior seems questionable. This unsexy work…

> So what happens is people convince themselves that the existing tech sucks and use that to rationalize doing the rewrite.

That certainly is a thing that happens, but you could use that to dismiss any technology at all. In the case of Kubernetes, it makes operations a lot easier to the (important) effect that the development teams can do a lot of their own operations work. This is important since they're the ones who are empowered to solve operations problems and it also eliminates the blame game between ops and dev. Further, it eliminates a lot of coordination with a separate ops team--the dev teams aren't competing to get time from an ops team; they can solve their own problems, especially the most common ones. This also has the nice property of freeing the SREs to work on high-level automation, including integrating tools from the ecosystem (e.g., cert-manager, external-dns, etc).

Kubernetes certainly isn't the final stage in the evolution, but it's a welcome improvement.

Re: Google admits Kubernetes container tech is too complex

#349

Post by throwaway894345 »

anchochilis wrote: post #254
z9e wrote: post #233

Earlier quoted context omitted.

... Until they hire you back because entropy exists :P I'm on the DevOps side as well going through the same transition, k8s also allows insane customization, and I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating. The k8s scene seems to be filled with constant scope creep and refactoring to get it just…

Ha, are you me? I really pushed for us to follow the "change-as-little-as-possible and ship to prod quickly" route. Prod is where things get hard, and it's better to find out what's hard sooner rather than later. We are running a handful of stateful services in K8s (things like MongoDB for which GCP doesn't have a compelling and affordable managed offering). It's definitely more complex than transitioning a stateless…

> We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy.

Agreed. The on-prem story is still really messy, but I think there's a lot of third-party work to build on-prem distributions that are cut and dry. Unfortunately, there are lots of them right now and it's not clear what the advantages and pitfalls are of each. Things will settle and this problem will be solved with time, but for now it's quite a pain point.

Re: Google admits Kubernetes container tech is too complex

#350

Post by joshuamorton »

bob1029 wrote: post #176

I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…

Horizontal scaling brings down total compute use if you have many distinct uses.

Imagine of your only unit of compute is a single bulky machine. You don't fully saturate it, but you need a second machine to avoid downtime anyway. Now you spin up a second or third service and suddenly you need 5 or ten machines and your compute utilization is 20%. You can pack things in tighter. But then you have a knapsack problem, and that's easier to solve efficiently with many small blocks even if it costs you a 1% overhead or whatever.

Post reply on HN