As Kubernetes grows, a startup ecosystem develops in its wake

I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S.

I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster.

My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore.

I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...)

The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.

Borg is an accumulation of a decade's work with containers at Google, and has been described by googlers as a rich but a little messy, having been designed incrementally over many years as needs have surfaced. Borg could never be open-sourced because it's so specific to Google; for example, it uses Google's own cgroups-based container tech, not Docker/OCI/etc. Omega, as I understand, was an effort to clean up Borg and modernizing it, but apparently it was never put into production; instead, some of the innovations ended up being backported to Borg [1].

More importantly, Kubernetes is based roughly on the same design as Borg: A declarative, consistent object store, with controllers, schedulers and other bits and pieces orchestrating changes to the store, mediated by a node-local controller (Borglet/Kubelet). A major difference between Borg and Kubernetes is that with Borg, the object store is exposed to clients, whereas Kubernetes hides it behind an API. Another difference is the structure of containers; Borg's "allocs" are coarser-grained than pods and Borg is less strict about where things go, which googlers have described as a shortcoming compared to Kubernetes' strict pod/container structure. Another difference, also seen as a shortcoming, is that Borg lacks Kubernetes' one-IP-per-pod system; all apps on Borg apparently share the host's network interface. Kubernetes also innovates on Borg in several ways; for example, Borg doesn't have labels [2].

Borg, from what I gather, scales much further than Kubernetes at this point, but it's really not related to the design. The design is fundamentally the same.

Yegge's criticisms are too handwavy ("overcomplicated") to counter, but I don't think Yegge knows what he's talking about here. As for "benefit": Not sure what you mean by this, but Kubernetes arguably comes with benefits — declarative ops, platform abstractions, container isolation — even if you're just running a single node. The notion that you only need Kubernetes if you have "Google-scale problems" is just nonsense.

PS. What's "Grab"?

[1] https://ai.google/research/pubs/pub44843 (I recommend reading this paper)

[2] https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...

According to googlers here on HN, Google does use Kubernetes internally via GCP/GKE. But clearly those apps are in the minority, given Google's huge investment in Borg.