Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borg: The Next Generation [pdf]

eurosys2020.org

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Borg: The Next Generation [pdf]

#21

Post by polskibus »

jeffbee wrote: post #18
polskibus wrote: post #10

How much different/better/worse is this compared to Kubernetes?

k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5…

Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?

Re: Borg: The Next Generation [pdf]

#22

Post by lokar »

polskibus wrote: post #21
jeffbee wrote: post #18

Earlier quoted context omitted.

k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5…

Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?

It's not worse overall, just different.

Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization.

K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways.

For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).

Re: Borg: The Next Generation [pdf]

#23

Post by star-trek-fleet »

sandGorgon wrote: post #17
justicezyx wrote: post #4

Was involved in this research project for a few months in early 2019. Feel free to post some questions that are not sensitive to internal details, and I can answer them here. Edit: not one of the coauthors, since I left the team in April 2019.

How do people who work on Borg look at kubernetes? Is it the real "next generation"

When comparing Borg vs. K8s:

* Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s.

* Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications.

* K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings.

In summary:

* K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production.

* As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing.

Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.

Re: Borg: The Next Generation [pdf]

#24

Post by justicezyx »

pwg wrote: post #20

Also a name clash with Borg the backup utility (which is the #5 item returned from a google search for "borg" for me when I just tested what would come back): https://borgbackup.readthedocs.io/en/stable/

Nah, Borg as a cluster manager way cooler and appropriate than Borg the backup utility.

Re: Borg: The Next Generation [pdf]

#25

Post by justicezyx »

GauntletWizard wrote: post #8
kgraves wrote: post #3

Interesting, I can only wish to run this sort of tech for my own small company.

You can! I have all the equivalent graphs and most of the equivalent features on the k8s clusters I run. Google has commoditized borg, at least for the scales that most companies will run.

Not really, Borg is itself only at the scale it operates at. It's like Michael Jordan is not the baseball god, if it's only allowed to playing with 5 year old kiddos.

Re: Borg: The Next Generation [pdf]

#28

Post by jeffbee »

polskibus wrote: post #21
jeffbee wrote: post #18

Earlier quoted context omitted.

k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5…

Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?

That seems a bit unfair. The community is getting a pretty square deal with k8s, few people need to manage cells with more than 5000 machines, and those that do can probably afford to write their own schedulers that meet their scalability and performance needs. Google can't just throw Borg over the wall because it's pretty hairy, relies on other parts of Google that also aren't public, and because of the general belief that "the community" is not ready to deal with a large, high-performance C++ program of the type that Google tends to write, which is partly condescension but also pretty much true.

Re: Borg: The Next Generation [pdf]

#30

Post by altgoogler »

pwg wrote: post #20

Also a name clash with Borg the backup utility (which is the #5 item returned from a google search for "borg" for me when I just tested what would come back): https://borgbackup.readthedocs.io/en/stable/

Googler here.

Borg @ Google predates Borg the backup utility by several years.

Post reply on HN