31–40 of 61 posts
Is Borg essentially Google's version of Condor?
What is condor?
How much different/better/worse is this compared to Kubernetes?
Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters v…
How much different/better/worse is this compared to Kubernetes?
Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters v…
CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those.
For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment).
You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.)
My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.
Also a name clash with Borg the backup utility (which is the #5 item returned from a google search for "borg" for me when I just tested what would come back): https://borgbackup.readthedocs.io/en/stable/
- The Borg are a fictional race of aliens in Star Trek: Next Generation. They were first shown on TV in 1989. This appears to be the basis for the software projects' naming.
- Borg, Google's cluster management system, was introduced at some point prior to 2006 (per [0], published in 2016, which states Borg was introduced "over a decade ago").
- Borg the backup system was forked from Attic in 2015 [1]
[0]: https://cloud.google.com/blog/products/gcp/from-google-to-th...
[1]: https://borgbackup.readthedocs.io/en/stable/changes.html#ver...
Earlier quoted context omitted.
Kubernetes is very different from Borg.
Can you share how? I’ve heard of borg previously as a inspiration for open source k8s but I’m surprised to hear that it’s very different.
However, the developer experience is hugely different, both in the things that Kubernetes does have (Configuration Canaries) and the things it doesn't have (Global Load Balancing). There's patterns and hacks around most of these, but they're often created by developers who didn't understand the tradeoffs.
Most importantly: Borg is very well sanded. Borg was a production-ready system ten years ago, and while it's had to grow and change, there's many things that "work together" just nicely in the Google Ecosystem, that just don't yet outside. Partially because they support many more alternative implementations that don't quite have a standard API, but mostly because there's just rough spots about how you deal with marginal/edge cases that aren't found and documentation written on how to avoid them.
Earlier quoted context omitted.
How do people who work on Borg look at kubernetes? Is it the real "next generation"
When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integ…
It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.
Earlier quoted context omitted.
You can! I have all the equivalent graphs and most of the equivalent features on the k8s clusters I run. Google has commoditized borg, at least for the scales that most companies will run.
Not really, Borg is itself only at the scale it operates at. It's like Michael Jordan is not the baseball god, if it's only allowed to playing with 5 year old kiddos.
I recently got back on a project that was on maintenance mode for a year - But because it had a nice K8s setup, it had more or less managed itself through OS upgrades and rebuilds with very little user intervention.
Does Borg also track and allocate network usage? That has been an issue for me in compute environments in the past for high IO situations.
How much different/better/worse is this compared to Kubernetes?
k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5…
To that extent, there's a hole left dealing with things that do need to consider state versus run networked API services. Kubernetes seems to have no good story here, whether it's the evolutionary progress happening around StatefulSets/PVC/PV, or a per appliance operator for you, and you, and you, which punch a hole as big as you like in the scheduling abstraction. Streaming for example is a notable pain, but pretty much every OSS project created in this century that has state needs a compensating tool, typically an operator, to function on Kubernetes. I'm not even sure at this point whether state is a design consideration that can be retrofitted—that's not a criticism of Kubernetes, but it is a complication and investment factor for stateful workloads. So what may happen should Kubernetes be one of those things that does in fact end up being a long term technology, is the entire software industry offloads state management to vendors (ie to a handful of cloud services), or something in open source reacts and is created to fill the infrastructure gap for state management.