Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Kubernetes Kustomize KEP Kerfuffle

gravitational.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: The Kubernetes Kustomize KEP Kerfuffle

#21

Post by marcc »

mfer wrote: post #16

Since I was at the core of what the article wrote about I feel the need to jump in. Some of the quotes are from me, after all. kustomize vs Helm is a bit if an improper comparison. Kustomize and the ways to use kubectl are often about configuration management and Helm is about package management. This is an important difference. For example, take Ansible doing configuration management and using Helm to install a pack…

Helm is a good package management tool for application level configuration items. The possible configuration options that a cluster operator might want to configure is fixed. The application packager knows what they may be and can template them into environment variables or write them to configmaps. It doesn't matter how they are delivered, the values.yaml is a great place to collect this data, and Helm manages it nicely.

However, when it comes to cluster operators, or "runtime configuration", the application packager cannot possibly know all possible configuration changes that a cluster operator might want to make. Some are obvious, and Helm ecourages putting them into values.yaml. For example, most charts include a way to change resource limits or imagepullpolicy. But what if a cluster operator wants to change a config map to be deployed as a bitnami SealedSecret, because it's more secure and more compatible with the infrastructure? Or what if a cluster operator wants to deploy the Cloudflare Argo Tunnel Ingress controller which requires more: not only an annotation, but also additional fields. This cluster operator often ends up forking the Helm chart or deploying these parts out-of-band. Differentiating between runtime config and application config is why both Kustomize and Helm are both useful, together.

I like Helm as a package manager. But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included?

I like the sandbox projects and how the CNCF is managing them. I'm actively working on a different project (Open Policy Agent), and it's much younger than Helm or Kubectl. I agree with your concerns about the KEP process, and how it should be more well defined and followed at all times.

Re: The Kubernetes Kustomize KEP Kerfuffle

#22

Post by geerlingguy »

marcc wrote: post #21
mfer wrote: post #16

Since I was at the core of what the article wrote about I feel the need to jump in. Some of the quotes are from me, after all. kustomize vs Helm is a bit if an improper comparison. Kustomize and the ways to use kubectl are often about configuration management and Helm is about package management. This is an important difference. For example, take Ansible doing configuration management and using Helm to install a pack…

Helm is a good package management tool for application level configuration items. The possible configuration options that a cluster operator might want to configure is fixed. The application packager knows what they may be and can template them into environment variables or write them to configmaps. It doesn't matter how they are delivered, the values.yaml is a great place to collect this data, and Helm manages it ni…

This is exactly why I’m not using Helm for most applications; it would be more maintenance to try to maintain all the more specific forked helm charts and a helm chart repository than to maintain the direct and lightly templated manifests which allow me to have 100% immutable definitions of all my infrastructure (like I prefer with or without Kubernetes), but without also maintaining a huge set of more complicated Helm charts.

Re: The Kubernetes Kustomize KEP Kerfuffle

#23

Post by [deleted] »

gouggoug wrote: post #18
justicezyx wrote: post #6

Earlier quoted context omitted.

You missed the key part. This is why it was open sourced, as a balance between control leverage and adoption.

> This is why it was open sourced, as a balance between control leverage and adoption. I feel like I read this kind of statement all the time and I don't understand the reasoning behind it. Google came up with k8s. They could have kept it for themselves, or, open source it. They open-sourced it. It definitely benefits them, but, it also benefits us a lot. Saying they "open sourced, as a balance between control levera…

[deleted]

Re: The Kubernetes Kustomize KEP Kerfuffle

#24

Post by mfer »

marcc wrote: post #21
mfer wrote: post #16

Since I was at the core of what the article wrote about I feel the need to jump in. Some of the quotes are from me, after all. kustomize vs Helm is a bit if an improper comparison. Kustomize and the ways to use kubectl are often about configuration management and Helm is about package management. This is an important difference. For example, take Ansible doing configuration management and using Helm to install a pack…

Helm is a good package management tool for application level configuration items. The possible configuration options that a cluster operator might want to configure is fixed. The application packager knows what they may be and can template them into environment variables or write them to configmaps. It doesn't matter how they are delivered, the values.yaml is a great place to collect this data, and Helm manages it ni…

Ya know what's great about debating tools? Because we have a separation, different things can innovate at different rates and people can do different things.

You can move from deploying to AWS to Kubernetes all while your CI pipeline and config management stay the same. Maybe at another time you change your CI system.

In all of this we can even have tool fights. Jenkins vs CircleCI. vim vs emacs. And new players can come along like vscode.

Different departments in the same companies, sometimes billion dollar companies, can even do this.

This is one of the reasons I personally like a separation of concerns with projects.

There is one thing about Helm...

> But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included?

I don't think so. Without being long winded, this is about user experience and encapsulations. Kubernetes is hard to learn. Then do learn the business logic for installing all the things you use. To use the old wordpress example, should someone installing wordpress into a cluster need to know the business needs of MySQL to create the k8s config for it? Most app devs don't want to learn all the k8s object configs... it's a lot to learn.

As for CRDs, that's about dependency management.

But, if we make Kubernetes really hard, with lots of CRDs we have to start to deal with what CRDs are in what clusters? And, what experience does that provide for end users? How do we not make the UX barrier to entry to high for the people who need to use it?

Are users interested in Kubernetes or are they interested in their apps and core business logic?

Re: The Kubernetes Kustomize KEP Kerfuffle

#25

Post by mfer »

gouggoug wrote: post #18
justicezyx wrote: post #6

Earlier quoted context omitted.

You missed the key part. This is why it was open sourced, as a balance between control leverage and adoption.

> This is why it was open sourced, as a balance between control leverage and adoption. I feel like I read this kind of statement all the time and I don't understand the reasoning behind it. Google came up with k8s. They could have kept it for themselves, or, open source it. They open-sourced it. It definitely benefits them, but, it also benefits us a lot. Saying they "open sourced, as a balance between control levera…

Why did Google create Kubernetes and open source it? Why did the people who created Kubernetes all leave Google? What was the business strategy behind it all?

After all, the created omega but it never did replace borg. Kubernetes wasn't meant to replace borg, was it?

Google didn't create and release Kubernetes out of the goodness of their hearts. Have you ever pondered the strategy behind it all? Maybe they couldn't have kept it to themselves because that would have defeated the underlying purpose in the first place?

Re: The Kubernetes Kustomize KEP Kerfuffle

#26

Post by aberoham »

mfer wrote: post #17
aberoham wrote: post #2

Author here. This was an extremely hard topic to write about. Happy to answer questions but really hoping those from sig-cli and sig-architecture chime in here. Is this Kerfuffle the root of the sudden resurgence of KEPs?

Correction for your post, it's a CNCF Technical Oversight Committee (TOC) rather than a Kubernetes one. The Kubernetes steering committee oversees Kubernetes. The CNCF TOC oversees all the CNCF projects (Prometheus, Kubernetes, Helm, etc).

Interesting quibble, will edit.

Re: The Kubernetes Kustomize KEP Kerfuffle

#27

Post by mgoodness »

the_duke wrote: post #10
aberoham wrote: post #2

Author here. This was an extremely hard topic to write about. Happy to answer questions but really hoping those from sig-cli and sig-architecture chime in here. Is this Kerfuffle the root of the sudden resurgence of KEPs?

Question: one of the main advantages of helm (for me) is the pruning of dead resources. As in, my chart contains 3 services , I remove one of them including some related things like secrets, configmaps, etc and deploy, helm will clean up everything redundant. Does Kustomize handle this problem?

It does not. Kustomize only generates the manifests. Typically you'd use a GitOps-driven CD application (ArgoCD, Weave Flux) to create/change/delete resources.

Re: The Kubernetes Kustomize KEP Kerfuffle

#28

Post by aberoham »

justicezyx wrote: post #12
aberoham wrote: post #2

Author here. This was an extremely hard topic to write about. Happy to answer questions but really hoping those from sig-cli and sig-architecture chime in here. Is this Kerfuffle the root of the sudden resurgence of KEPs?

I'd like you to start a series on less-formal activities surrounding the k8s community. For example, public events, new major adopters, war story of using k8s that affect k8s' fundamental technical philosophy, etc.

Gravitational blog posts[1] I've participated in are generally in these topic areas. Any more specific requests?

[1] https://gravitational.com/blog/

Re: The Kubernetes Kustomize KEP Kerfuffle

#29

Post by thockingoog »

mfer wrote: post #25
gouggoug wrote: post #18

Earlier quoted context omitted.

> This is why it was open sourced, as a balance between control leverage and adoption. I feel like I read this kind of statement all the time and I don't understand the reasoning behind it. Google came up with k8s. They could have kept it for themselves, or, open source it. They open-sourced it. It definitely benefits them, but, it also benefits us a lot. Saying they "open sourced, as a balance between control levera…

Why did Google create Kubernetes and open source it? Why did the people who created Kubernetes all leave Google? What was the business strategy behind it all? After all, the created omega but it never did replace borg. Kubernetes wasn't meant to replace borg, was it? Google didn't create and release Kubernetes out of the goodness of their hearts. Have you ever pondered the strategy behind it all? Maybe they couldn't…

> Why did the people who created Kubernetes all leave Google?

Umm, we didn't?

> created omega but it never did replace borg

It did have material impact on Borg.

Sometimes the strategy is simply "we know how to do this, and we'd prefer not to see you go through all the same pain to figure it out".

Re: The Kubernetes Kustomize KEP Kerfuffle

#30

Post by mfer »

thockingoog wrote: post #29
mfer wrote: post #25

Earlier quoted context omitted.

Why did Google create Kubernetes and open source it? Why did the people who created Kubernetes all leave Google? What was the business strategy behind it all? After all, the created omega but it never did replace borg. Kubernetes wasn't meant to replace borg, was it? Google didn't create and release Kubernetes out of the goodness of their hearts. Have you ever pondered the strategy behind it all? Maybe they couldn't…

> Why did the people who created Kubernetes all leave Google? Umm, we didn't? > created omega but it never did replace borg It did have material impact on Borg. Sometimes the strategy is simply "we know how to do this, and we'd prefer not to see you go through all the same pain to figure it out".

Tim, I should not have used "all" there.
Post reply on HN