Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borg: The Next Generation [pdf]

eurosys2020.org

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Borg: The Next Generation [pdf]

#41

Post by sterlind »

star-trek-fleet wrote: post #23
sandGorgon wrote: post #17

Earlier quoted context omitted.

How do people who work on Borg look at kubernetes? Is it the real "next generation"

When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integ…

There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg besides Google. I find it sad though, because I fell in love with it from the rumors I heard and papers I read when I redesigned Azure's cluster scheduler algorithm.

I think it's sad that 99% of the resources are used by hogs, though. I always thought it'd be neat to build on tiny containers; this suggests that you really can be fine with a few giant tenants and minimal colocation.

sigh some day I'll get to design my dream Borg successor, even if it's a TempleOS-like art project after I've descended into dementia.

Re: Borg: The Next Generation [pdf]

#42

Post by jeffbee »

dehora wrote: post #40
jeffbee wrote: post #18

Earlier quoted context omitted.

k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5…

Yes for sure, in terms of expansion/adoption. It's not so certain in terms of function/utility. As a Google outsider, gRPC really does seems like Stubby for the rest of us (with balancing left as tradeoff for the community). Kubernetes does not seem functionally at all, to be a Borg/Omega. it's more like a porcelain for running Heroku/12-factor/Nanoservice style workloads on top of a Borg-like (that's no small thing,…

I don't think you can count on Google solving stateful for k8s, because within Google all storage devices and data thereupon are, to a fair approximation, totally disposable. There is nothing at Google considered a "stateful service" the way k8s community members mean it, e.g. a mysql server with critical local files.

In my personal opinion it is more valuable to adopt the Google model where no local file is considered critical, than it is to try to cram statefulness into an otherwise cloud-native stack. I feel that if you still care about specific files on specific disks then you really haven't fully adopted the meaning of cloud-native.

Re: Borg: The Next Generation [pdf]

#43

Post by justicezyx »

sterlind wrote: post #41
star-trek-fleet wrote: post #23

Earlier quoted context omitted.

When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integ…

There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg…

Mind to connect me at info@nascentcore.ai?

We might have something to show.

Re: Borg: The Next Generation [pdf]

#46

Post by estebarb »

jeffbee wrote: post #42
dehora wrote: post #40

Earlier quoted context omitted.

Yes for sure, in terms of expansion/adoption. It's not so certain in terms of function/utility. As a Google outsider, gRPC really does seems like Stubby for the rest of us (with balancing left as tradeoff for the community). Kubernetes does not seem functionally at all, to be a Borg/Omega. it's more like a porcelain for running Heroku/12-factor/Nanoservice style workloads on top of a Borg-like (that's no small thing,…

I don't think you can count on Google solving stateful for k8s, because within Google all storage devices and data thereupon are, to a fair approximation, totally disposable. There is nothing at Google considered a "stateful service" the way k8s community members mean it, e.g. a mysql server with critical local files. In my personal opinion it is more valuable to adopt the Google model where no local file is consider…

How Google handles disk storage? From what I remember of the papers, GFS depends on a disk service, that exposes the disk resources. And how YouTube store MySQL data? Uses GFS or some other mechanism?

Re: Borg: The Next Generation [pdf]

#48

Post by remus »

The resource usage section at the end was really interesting, and surprising to me. 1% of jobs use 99% of resources! It would be interesting to try and understand how this pattern came about and if there's particular engineering decisions that tend to lead to this situation where you have a handful of incredibly resource intensive jobs and loads of very lightweight jobs.

Re: Borg: The Next Generation [pdf]

#49

Post by enitihas »

estebarb wrote: post #46
jeffbee wrote: post #42

Earlier quoted context omitted.

I don't think you can count on Google solving stateful for k8s, because within Google all storage devices and data thereupon are, to a fair approximation, totally disposable. There is nothing at Google considered a "stateful service" the way k8s community members mean it, e.g. a mysql server with critical local files. In my personal opinion it is more valuable to adopt the Google model where no local file is consider…

How Google handles disk storage? From what I remember of the papers, GFS depends on a disk service, that exposes the disk resources. And how YouTube store MySQL data? Uses GFS or some other mechanism?

Not worked at Google, but from what I read, only GFS or Colossus worry about disk storage, every other application can write only to GFS and have no dependency on local disks.

Re: Borg: The Next Generation [pdf]

#50

Post by thu2111 »

sterlind wrote: post #41
star-trek-fleet wrote: post #23

Earlier quoted context omitted.

When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integ…

There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg…

Don't be fooled by the 99% figure. It says in 2011 the figure wasn't much different. Well that was a bit surprising to me because when I was using Borg in 2011 it didn't feel that way.

Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs.

The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc.

The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc.

So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.

Post reply on HN