Kubernetes SidecarContainers feature is merged • Hacker News Board

Re: Kubernetes SidecarContainers feature is merged

#41

Post by birdyrooster » Mon, Jul 10, 2023, 6:39 PM UTC

jauntywundrkind wrote: post #27

penciltwirler wrote: post #25

Earlier quoted context omitted.

There's lots of tools built on top of K8s to accomplish this tho. For example, Argo, Tekton, Flyte etc.

Absolutely, no shortage of things atop. Helm is probably the most well used composition tool. It seems unideal to me to forever bunt on this topic, leaving it out of core forever. Especially when we are slowly adding im very specialized composition orchestration tools in core.

Requiring the user to write their own operators to manage state using the kubernetes api is very much a feature and not something which is missing.

Re: Kubernetes SidecarContainers feature is merged

#42

Post by annexrichmond » Mon, Jul 10, 2023, 7:19 PM UTC

The lack of native sidecar support was my biggest surprise when moving from ECS to EKS, and it was not fun hacking with shared process IDs to accomplish sidecars. I'm glad this is finally in but also curious how it takes roughly 3ish years(?) from KEP proposal to merge?

Re: Kubernetes SidecarContainers feature is merged

#43

Post by theptip » Mon, Jul 10, 2023, 7:24 PM UTC

jauntywundrkind wrote: post #24

On the one hand, great. The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about defining pieces of state, but managing blocks of state & multiple things at once is left almost entirely to external tools. The ability to compose &sequence multiple containers feels like a very specific example of a much broader general capability. There's bed…

This is all true, and if you read the KEPs they were thinking about this. One camp was advocating for solving the problem of specifying the full dependency graph spec (of which sidecars are one case), another advocating for just solving the most needed case with a sidecar-specific solution to get a solution shipped. The latter was complicated by a desire to at least leave the door open for the former.

Pragmatism won out, thankfully IMO.

Edit to add: see this better description from one of the senior k8s maintainers: https://news.ycombinator.com/item?id=36666359

Re: Kubernetes SidecarContainers feature is merged

#44

Post by tmzt » Mon, Jul 10, 2023, 7:32 PM UTC

Is there a clean way to share an emptyDir between sidecar(s) and main container(s)?

Looking at the logging usecase and want to be able to add a log shipper sidecar to a pod with ephemeral storage.

Re: Kubernetes SidecarContainers feature is merged

#45

Post by sargun » Mon, Jul 10, 2023, 7:52 PM UTC

This is great. My team at Netflix (I'm not longer there) sponsored some of the work behind this, via Kinvolk (now acquired by MSFT). Great to see that it finally shipped. At the time, this was a blocker to us using Kubelet, and we thought it might take a few...months to sort out. Turns out it was closer to a few years, but its a tricky API, and important to get right.

Re: Kubernetes SidecarContainers feature is merged

#46

Post by 0xbadcafebee » Mon, Jul 10, 2023, 7:53 PM UTC

nrmitchi wrote: post #35

While this is a very welcome improvement in terms of functionality, I can't help by feel that the re-use of "restartPolicy" to mean something similar, but different, when used in a different context, is a very poor decision. Kubernetes already has an issue with having a (perceived) high barrier to entry, and I'm not sure that "restartPolicy on a container means this, unless isn't used in this list of containers, in w…

It's par for the course. Most of K8s's design has been shoving whatever crap they feel like in, regardless of confusion, difficulty, complexity, etc for the end user.

Re: Kubernetes SidecarContainers feature is merged

#47

Post by 0xbadcafebee » Mon, Jul 10, 2023, 7:59 PM UTC

politelemon wrote: post #34

NathanKP wrote: post #30

Earlier quoted context omitted.

Yeah I work on the team that builds Amazon Elastic Container Service so I can't help but compare this implementation with how we solved this same problem in ECS. Inside of an ECS task you can add multiple containers and on each container you can specify two fields: `dependsOn` and `essential`. ECS automatically manages container startup order to respect the dependencies you have specified, and on shutdown it tears th…

As a consumer, ECS + Fargate is my happy path. I appreciate the lack of complexity. Thanks.

Ditto. ECS/Fargate has always been the easiest, most flexible, most useful containerization solution. It's the one AWS service with the most value to containerized services, and the least appreciated.

Re: Kubernetes SidecarContainers feature is merged

#48

Post by 0xbadcafebee » Mon, Jul 10, 2023, 8:08 PM UTC

jauntywundrkind wrote: post #24

On the one hand, great. The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about defining pieces of state, but managing blocks of state & multiple things at once is left almost entirely to external tools. The ability to compose &sequence multiple containers feels like a very specific example of a much broader general capability. There's bed…

Orchestration of k8s wouldn't be necessary if they had made K8s' operation immutable. As it stands now you just throw some random YAML at it and hope for the best. When that stops working, you can't just revert back to the old working version, you have to start throwing more crap at it and running various operations to "fix" the state. So you end up with all these tools that are effectively configuration management tools to continuously "fix" the cluster back to where you want it.

I hope the irony is lost on no one that this is an orchestration tool for an immutable technology, and the orchestrator isn't immutable.

Re: Kubernetes SidecarContainers feature is merged

#49

Post by verst » Mon, Jul 10, 2023, 9:04 PM UTC

AtNightWeCode wrote: post #32

When I first learned about the sidecar pattern I thought it was great. I am not sure about it anymore. Most of it could be propagated to custom images or layers at the boundary. To me this feels a bit sketchy. Too have containers that kinda is part of the mesh but then does not share the same lifecycle as the mesh.

If you create a custom image you would need to create a complex health endpoint that is essentially only considered healthy if all the components baked into your image are considered healthy. This gets harder when you are not the author of the sidecar process on which you rely. With a single image it would be easier to run into a situation where the sidecar process (baked into your image) is in an unhealthy state but your container is not restarted because the app itself is not reporting unhealthy status.

Re: Kubernetes SidecarContainers feature is merged

#50

Post by aranelsurion » Mon, Jul 10, 2023, 9:06 PM UTC

yla92 wrote: post #12

A very welcome change. It's gonna be helpful for the case where the database proxy (CloudSQL) and the main container got terminated out of order. https://cloud.google.com/sql/docs/postgres/connect-kubernete...

Just FYI for people who don't know about it yet: with cloudsql-proxy v2 there's a new parameter called "--quitquitquit" that starts up an HTTP endpoint to be used for graceful shutdowns. Basically your main container makes a POST to this endpoint, and sidecar exits.