It's so saddening to see how the Kubernetes hype-cycle follows OpenStack and all the fundamental problems still seem unsolved. I sometimes feel like its just the same story playing out 5 years later, one layer up to the stack (IaaS -> CaaS) and with other fools to fall for it (with OpenStack it was sysadmins trying to run a control plane, with Kubernetes its devs trying to run infrastructure).
The abstractions we have available to build and run distributed systems may have improved, but they still suck in the grand scheme of things. My personal nightmare is that nothing better comes along soon.
> - Is it the networking model that is simple from the consumption standpoint but has too many moving parts for it to be implemented?
Many poor sysadmins before us have tried to implement Neutron (OpenStack Networking Service) with OvS or a bunch of half-assed vendor SDNs. Or LBaaS with HAProxy.
> - Is it the storage model, CSI and friends?
I mean, the most popular CSI for running on-premise is rook.io, which is just wrapping Ceph. Ceph is just as hard to run as ever, and a lot of that is justified by the inherent complexity of providing high performance multi-tenant storage.
> - Is it the bunch of controller loops doing their own things with nothing that gives a "wholesome" picture to identify the root cause?
Partially. One advantage the approach has is that it's conceptually simple, consistent and feels easy to compose complex behavior. The problem is that Kubernetes enforces very little structure, even basics like object ownership. The result is unbounded complexity. A lack of tooling (e.g. time travel debugging for control loops) makes debugging complex interactions next to impossible. This is also not surprising, control loops are a very hard problem and even simple systems can spiral (or oscillate) out of control very quickly. Control theory is hard. David Anderson has a pretty good treatise of the matter https://blog.dave.tf/post/new-kubernetes/
Compared to OpenStack, Kubernetes uses a conceptually much simpler model (control loops + CRDs) and does a much better job at enforcing API consistency. Kubernetes is locally simple and consistent, but globally brittle.
The downside is that it needs much more composition of control loops to do meaningful work, and that leads to exploding complexity because you have a bunch of uncoordinated actors (control loops) each acting on partial state (a subset of CRDs).
The implementation model of an OpenStack service otoh is much simpler because they use straight forward "workflows", working on a much bigger picture of global state, e.g. neutron owning the entire network layer. This makes composition less of a source for brittleness, not that OpenStack still has its fair share of that as well. Workflows are however much more brittle locally, because they cannot reconcile themselves in case things go wrong.