i can't help but very lol @ kubernetes as the successor to borg :0
Into the Borg – SSRF inside Google production network
71–79 of 79 posts
Re: Into the Borg – SSRF inside Google production network
#72Earlier quoted context omitted.
What's the size of the transitive dependency graph of Borg? 10MLOC? 50MLOC? 100MLOC? I have no idea. But it's a lot of code no matter what. Open sourcing that much code is a huge undertaking, unless you're just planning to throw it over the wall with no expectation of external people working on it. On the other hand starting from scratch you get to grow the community and the codebase in lockstep.
It may be a large undertaking but yes, it's clearly still less work to release code that exists and build a community around it, than rewrite it all from scratch and also build a community around that too.
You just filled up my BINGO card by saying "clearly" to indicate that you have no idea what you are talking about.
Re: Into the Borg – SSRF inside Google production network
#73Earlier quoted context omitted.
Dumb question: why does K8s use a centralized architecture like Borg, if the perf gains from an Omega-style shared-state scheduler decentralization (and maybe a Mesos-style two-level scheduler for batch with multiple frameworks) were already known, and Omega was already being folded back into Borg? Is this related to (I'm assuming) the fact that K8s was originally architected "mostly" with service rather than batch i…
You can actually build an Omega vertical / Mesos framework architecture on Kubernetes, as described in this doc[1]. That doc pre-dated CRDs; the way you'd do it today is to build the application lifecycle management part of the framework using a CRD + controller, and run an application-specific scheduler (for pods created by that controller) alongside the default scheduler. The Kubernetes documentation page explainin…
> Borg only worked with a single scheduler
No love for rescheduler? =(
Re: Into the Borg – SSRF inside Google production network
#74Earlier quoted context omitted.
In this thread there is a repeated meme of "Borg is way more scalable than Kubernetes, and will always be so". But this ignores a lot of the history of Borg. When Borg was first created, it was not nearly as scalable as its current incarnation. We hit scalability bugs and limitations all the time! (I was working on a team which was exploring the scalability limits of MapReduce, which was often very good at finding th…
Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop. As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled o…
> It’s decisions regarding how the cluster maintains state that hamstring it
Jed, you keep repeating this like it's true, but it's not actually so. Here's an excerpt from Borg paper (which David co-authored btw ;-)):
> A single elected master per cell serves both as the Paxos leader and the state mutator, handling all operations that change the cell’s state, such as submitting a job or terminating a task on a machine.
And while we're at it, I don't know what it has to do with FauxMaster since it ran single replica and the passage about C++ is just pure fud.
Re: Into the Borg – SSRF inside Google production network
#75Earlier quoted context omitted.
You can actually build an Omega vertical / Mesos framework architecture on Kubernetes, as described in this doc[1]. That doc pre-dated CRDs; the way you'd do it today is to build the application lifecycle management part of the framework using a CRD + controller, and run an application-specific scheduler (for pods created by that controller) alongside the default scheduler. The Kubernetes documentation page explainin…
> Borg only worked with a single scheduler No love for rescheduler? =(
The rescheduler in Borg isn't a scheduler -- it just evicts pods, and then they go into the regular scheduler's pending queue and the regular scheduler decides where to schedule them. (At least that's how it worked at the time I left the project -- I assume it hasn't changed in this regard, but I don't know for sure.)
Because the name is confusing, we called the Kubernetes version of the Borg rescheduler the "descheduler" (https://github.com/kubernetes-incubator/descheduler) to make it clear that it doesn't actually schedule, just evicts. (There actually is something in Kubernetes called the "rescheduler" (https://kubernetes.io/docs/tasks/administer-cluster/guarante...) but it's a long story and we never should have named it that).
Re: Into the Borg – SSRF inside Google production network
#76Earlier quoted context omitted.
It may be a large undertaking but yes, it's clearly still less work to release code that exists and build a community around it, than rewrite it all from scratch and also build a community around that too.
You just filled up my BINGO card by saying "clearly" to indicate that you have no idea what you are talking about.
I've worked at Google for many years, and built open source communities based on code I've written from scratch several times. This is not an area I'm inexperienced in.
Re: Into the Borg – SSRF inside Google production network
#77Earlier quoted context omitted.
Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop. As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled o…
> It’s decisions regarding how the cluster maintains state that hamstring it Jed, you keep repeating this like it's true, but it's not actually so. Here's an excerpt from Borg paper (which David co-authored btw ;-)): > A single elected master per cell serves both as the Paxos leader and the state mutator, handling all operations that change the cell’s state, such as submitting a job or terminating a task on a machine…
Just curious, does Borgmaster use Chubby, or is it a completely separate Paxos store?
Re: Into the Borg – SSRF inside Google production network
#78Earlier quoted context omitted.
> It’s decisions regarding how the cluster maintains state that hamstring it Jed, you keep repeating this like it's true, but it's not actually so. Here's an excerpt from Borg paper (which David co-authored btw ;-)): > A single elected master per cell serves both as the Paxos leader and the state mutator, handling all operations that change the cell’s state, such as submitting a job or terminating a task on a machine…
Just curious, does Borgmaster use Chubby, or is it a completely separate Paxos store?
It’s using Chubby for locking (it’s actually next sentence in that Borg paper) and some othe things not related to quorum that i cant go into. This is different from kube master that uses etcd for everything but in terms of performance it’s not a big deal because elections dont happen often (and youd be surprised how many ppl run k8s with a single master setup, even GKE)
Re: Into the Borg – SSRF inside Google production network
#79I found this really cool to read into. Some of it went over my head but a great read none the less. Thanks for sharing.
Why was this downvoted? I really don't understand the users of this site sometimes.