Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a Kubernetes CRD Controller in Rust

technosophos.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Writing a Kubernetes CRD Controller in Rust

#21

Post by amerine »

parhamn wrote: post #3

How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space it started feeling like there were too many moving parts and a hectic mess. Stacks became more difficult to reproduce (controllers, their versions and platform limitations) and people moved too many things into custom resources. Did the custom resource controll…

The “kuberbetes as a reconciler loop” for CRDs pattern is its must useful quality to us at Heroku.

Re: Writing a Kubernetes CRD Controller in Rust

#22

Post by dilyevsky »

jacques_chester wrote: post #13
Thaxll wrote: post #12

Earlier quoted context omitted.

I also write CRD, but I don't beleive for one second that you have more than 10x less code, especially for generated code that you don't touch at all. It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that.

It's a different language with a different type system. I've also interacted with the API Server using a Ruby library that relied on metaprogramming. I could do things in a dozen lines that would require thousands of lines of checked-in Go. Replying to your edit: > It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keepin…

I don’t think openapi is necessary. All endpoint paths are easy to construct knowing gvk tuples which can be easily extracted from k8s.io/api using some go package analysis (you can also do it at runtime by parsing openapi endpoint). The data types are all defined in generated.proto in the same repo.

Re: Writing a Kubernetes CRD Controller in Rust

#23

Post by psoundy »

bogomipz wrote: post #15
jacques_chester wrote: post #9

Earlier quoted context omitted.

> How has Kubernetes being a giant reactive resource API panned out? It was never reactive, at least in the sense of the Reactive Manifesto. There is no system of backpressure. > Did the custom resource controllers pan out well? I think it will prove to be useful for a handful of uses, but that in general, it will wind up like Wordpress and Jenkins plugins. Powerful, popular, but a guaranteed mess. Kubernetes was not…

>"I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws." Could you elaborate on the "design flaws" part? Do you feel that the CRD model itself has design flaws or are you referring to specific project's CRDs?

tl;dr If you’re operating with CRDs at trivial scale, you probably having nothing to worry about. But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved.

——-

The usage patterns of native k8s types and the implications those patterns have on the scalability and reliability of etcd and the apiserver are relatively well-understood. CRDs can be a wild-card, though, and afaik testing efforts thus far have not investigated worst-case usage of CRD-based applications.

As commonly deployed, CRDs are served from the same apiservers and etcd cluster that serves the native types for a k8s cluster. That can result in contention between the CRDs supporting 3rd party additions to a cluster and the native types critical to the health of a cluster. This kind of contention has the potential to bring a cluster to its knees.

Efforts like priority and fairness seek to ensure that the apiserver can prioritize at the level of the API call. But that won’t prevent watch caches from OOM’ing the apiserver if excessive numbers of CRDs are present. The judicious use of quotas could head off the creation of an excessive number of objects, but it’s not just count that matters - the size of each resource is also a factor.

In theory, CRDs could be isolated from native types by serving them from an aggregated apiserver backed by a separate etcd cluster. afaik this not a supported configuration today, and even if it were the additional resources required to support it (especially the separate etcd cluster) may be prohibitive for many use cases.

Re: Writing a Kubernetes CRD Controller in Rust

#24

Post by dhruvt10 »

jsabo wrote: post #11
parhamn wrote: post #3

How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space it started feeling like there were too many moving parts and a hectic mess. Stacks became more difficult to reproduce (controllers, their versions and platform limitations) and people moved too many things into custom resources. Did the custom resource controll…

I'm not sure what you mean by too many things ended up as custom resources -- it's the recommended extension point and you can do a lot with them that you can't do with core types. Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pret…

Really interested to know about the challenges you faced in revisioning but couldn't find the exact thing, do you happen to know a source for understanding it more?

Re: Writing a Kubernetes CRD Controller in Rust

#25

Post by jacques_chester »

Arnavion wrote: post #17
jacques_chester wrote: post #13

Earlier quoted context omitted.

It's a different language with a different type system. I've also interacted with the API Server using a Ruby library that relied on metaprogramming. I could do things in a dozen lines that would require thousands of lines of checked-in Go. Replying to your edit: > It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keepin…

As the maintainer of the Rust bindings that the library used in the article (kube) is backed by, I can confirm that Kubernetes' openapi spec requires a lot of Kubernetes-specific handling to generate a good client that generic openapi generators do not provide. Yes, that includes all the other Kubernetes clients in github.com/kubernetes-client like Python and .Net too. See https://github.com/Arnavion/k8s-openapi/blob…

Truthfully, the fact that your bindings don't use OpenAPI generators has tempted me to learn Rust just so that I can escape their madness.

Re: Writing a Kubernetes CRD Controller in Rust

#26

Post by jacques_chester »

psoundy wrote: post #23
bogomipz wrote: post #15

Earlier quoted context omitted.

>"I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws." Could you elaborate on the "design flaws" part? Do you feel that the CRD model itself has design flaws or are you referring to specific project's CRDs?

tl;dr If you’re operating with CRDs at trivial scale, you probably having nothing to worry about. But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved. ——- The usage patterns of native k8s types and the implications those patterns have on the scalability and reliability of etcd and the apiserver are relatively well-understood. CRDs can be a wild-ca…

I agree with all of this, with one nitpick intended to self-aggrandise.

You can actually nominate particular types be stored in particular etcd servers -- GKE does this to put Events into a separate etcd from everything else.

However, it still has problems. Firstly, you can only define it for inbuilt types. Secondly, it's common for different objects to cross reference each other through objectRefs and the like, which behave badly when you effectively perform a join in the API server over multiple etcds.

Re: Writing a Kubernetes CRD Controller in Rust

#27

Post by hardwaresofton »

Go bills itself as simple, but Rust, despite it's complexity (which is hidden from you with libraries like serde and others) is simpler to use at the higher level.

Rust has a near-best-of-class type system, better abstractions, and runs on everything from embedded to the browser. Go has captured the hearts of sysadmin types and those looking for something just a step up from python (as evidenced by much of the devops code written in it -- which I'm very grateful for, I don't think I'd use kubernetes if it was in Java still), so I can see it growing in that field.

I don't see how Rust doesn't become the most important emerging systems language (C & C++ obviously aren't going anywhere just by virtue of being incumbents) of the next decade.

Re: Writing a Kubernetes CRD Controller in Rust

#28

Post by jacques_chester »

lima wrote: post #19
jacques_chester wrote: post #9

Earlier quoted context omitted.

> How has Kubernetes being a giant reactive resource API panned out? It was never reactive, at least in the sense of the Reactive Manifesto. There is no system of backpressure. > Did the custom resource controllers pan out well? I think it will prove to be useful for a handful of uses, but that in general, it will wind up like Wordpress and Jenkins plugins. Powerful, popular, but a guaranteed mess. Kubernetes was not…

Regarding multi-tenancy: The CRD schema is global but versioned, so it's annoying-but-manageable with multiple tenants. Controllers can be scoped to a namespace without requiring cluster-level permissions, no?

> The CRD schema is global but versioned, so it's annoying-but-manageable with multiple tenants.

It means every controller for a CRD winds up installing another webhook. And having to test a variety of orderings. It's hard to get right.

> Controllers can be scoped to a namespace without requiring cluster-level permissions, no?

The difficulty is that Kubernetes RBAC is good at expressing rules "Role 'foo' can perform operation 'list' on kind 'Deployment'". But it's less capable of saying something like "Role 'foo' can perform operation 'list' on kind 'Deployment' which were created from kind 'CoolerDeployment'". It's also hard to delegate something, along the lines of "Role 'foo' can delegate ('create' over kind 'Deployment') within namespace 'bar'".

I think dissatisfaction with PodSecurityPolicy will cause Rego to worm its way into the core architecture over the coming years. It'll then eventually crowd out RBAC because you can impose (sort of, more or less) arbitrary rules. But none will dare call it ABAC.

Re: Writing a Kubernetes CRD Controller in Rust

#29

Post by jacques_chester »

dilyevsky wrote: post #22
jacques_chester wrote: post #13

Earlier quoted context omitted.

It's a different language with a different type system. I've also interacted with the API Server using a Ruby library that relied on metaprogramming. I could do things in a dozen lines that would require thousands of lines of checked-in Go. Replying to your edit: > It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keepin…

I don’t think openapi is necessary. All endpoint paths are easy to construct knowing gvk tuples which can be easily extracted from k8s.io/api using some go package analysis (you can also do it at runtime by parsing openapi endpoint). The data types are all defined in generated.proto in the same repo.

It depends again on language. The ruby library I used exploited exactly this discoverability to create objects at runtime. It was awesome, except when time came to navigate my codebase. Where does the FooKind exist? How do I checkpoint it? Drat.

The generated codebases like client-go provide a great deal more navigability of code rather than APIs. It is the code that I actually interface with. I have my complaints about the necessity of code generation, but I do like having some kind of in-advance typing to help me figure things out.

Re: Writing a Kubernetes CRD Controller in Rust

#30

Post by psoundy »

jacques_chester wrote: post #25
Arnavion wrote: post #17

Earlier quoted context omitted.

As the maintainer of the Rust bindings that the library used in the article (kube) is backed by, I can confirm that Kubernetes' openapi spec requires a lot of Kubernetes-specific handling to generate a good client that generic openapi generators do not provide. Yes, that includes all the other Kubernetes clients in github.com/kubernetes-client like Python and .Net too. See https://github.com/Arnavion/k8s-openapi/blob…

Truthfully, the fact that your bindings don't use OpenAPI generators has tempted me to learn Rust just so that I can escape their madness.

Not sure why you’re being downvoted. OpenAPI schema generation is one of the most challenging elements of developing with CRDs. Anyone who is not an apimachinery SME is likely to struggle at some point to get their schema generated as expected.
Post reply on HN