Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Billion User Load Balancer [video]

youtube.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Building a Billion User Load Balancer [video]

#11

Post by zongitsrinzler »

acaloiar wrote: post #4

In 2001 when I was 14, I had no idea what scalability was or how anyone scaled websites. I ran a site with 20k users on a single sever with a single Pentium IV (1.4? Ghz) processor from a garage with no climate control and the fastest cable connection I could afford. At peak hours the processor finally succumbed to the load and I had to buy a second server. I was able to talk a local ISP into co-locating the servers…

This is gold.

But really, DNS load balancing can be really useful (and is so easy to set up).

Re: Building a Billion User Load Balancer [video]

#12

Post by scurvy »

Not trying to be a wet blanket, but there's nothing really groundbreaking in this talk. It's pretty basic stuff that everyone else running L3 networks already does.

I'd be much more interested in a talk that is solely on proxygen.

I'd also like to know their fragmentation rate is between the L4 and L7 LB's. I know what ours is, but I'm sure FB takes a lot more large uploads than we do. Fragmentation is the downside to using IPVS in a L3 network.

Re: Building a Billion User Load Balancer [video]

#13

Post by Blackthorn »

feld wrote: post #7
feld wrote: post #2

Surprised they use tinydns.

Why would I get downvoted for that? It's a serious concern. It's missing modern and often important features unless you use unofficial patchsets, and some missing entirely (NSEC3). And then there are all the forks like dbndns from Debian, N-DJBDNS, etc...

I can't actually downvote but I imagine you got downvoted because your comment reeked of bikeshedding (regardless of whether you intended it that way or not).

Re: Building a Billion User Load Balancer [video]

#14

Post by cmrx64 »

rroriz wrote: post #6
stann wrote: post #5

Can anybody summarize the talk for folks like me who for several reasons cannot watch the video?

I'd like to complement this comment and suggest that this practice should be applied to links that may be blocked to a grand part of us, people that are behind a firewall.

There are also accessibility reasons for not being able to watch a video. Let alone dedicating the time to it.

Re: Building a Billion User Load Balancer [video]

#15

Post by seasonedschemer »

scurvy wrote: post #12

Not trying to be a wet blanket, but there's nothing really groundbreaking in this talk. It's pretty basic stuff that everyone else running L3 networks already does. I'd be much more interested in a talk that is solely on proxygen. I'd also like to know their fragmentation rate is between the L4 and L7 LB's. I know what ours is, but I'm sure FB takes a lot more large uploads than we do. Fragmentation is the downside t…

Not sure if the talk covered it, but the core Proxygen framework is opensource: https://github.com/facebook/proxygen

Re: Building a Billion User Load Balancer [video]

#16

Post by mhuffman »

cmrx64 wrote: post #14
rroriz wrote: post #6

Earlier quoted context omitted.

I'd like to complement this comment and suggest that this practice should be applied to links that may be blocked to a grand part of us, people that are behind a firewall.

There are also accessibility reasons for not being able to watch a video. Let alone dedicating the time to it.

There are also those that have limited bandwidth available, or very expensive bandwidth, where a video is a luxury.

Re: Building a Billion User Load Balancer [video]

#18

Post by nocarrier »

scurvy wrote: post #12

Not trying to be a wet blanket, but there's nothing really groundbreaking in this talk. It's pretty basic stuff that everyone else running L3 networks already does. I'd be much more interested in a talk that is solely on proxygen. I'd also like to know their fragmentation rate is between the L4 and L7 LB's. I know what ours is, but I'm sure FB takes a lot more large uploads than we do. Fragmentation is the downside t…

This stuff seems basic at first glance, but it's not easy to get right, and you're discounting a ton of effort that went into getting feedback loops and demand prediction to work properly in Cartographer for example. Maybe it's easy to slap together L4/L7 LB and throw a DNS server in front of it at smaller scales, but it's a notable accomplishment to do this at Facebook's scale. Nothing out there approaches Facebook's request rates.

There have been detailed talks about Proxygen's architecture at various open houses in Seattle and Menlo Park, and they should have HTTP2 support working well soon. I'd expect there's going to be more detailed blog posts on Proxygen at some point. Or you can check out the source code too.

Fragmentation isn't really an issue for either Shiv or Proxygen. I don't think they will share the rate but it is very low.

Re: Building a Billion User Load Balancer [video]

#19

Post by nocarrier »

feld wrote: post #2

Surprised they use tinydns.

It's a simple, fast DNS server that is straightforward to extend if you need to. The genius of Cartographer is that the backend that calculates where to assign traffic is separate from the DNS server frontend. This let FB use a known-good DNS server that got the common DNS protocol stuff right, and gave them more time to focus on all the magic bits in Cartographer's backend.

DNSSEC was never something we were aiming for, so it's fine that TinyDNS doesn't support it out of the box.

Re: Building a Billion User Load Balancer [video]

#20

Post by nocarrier »

scoj wrote: post #3

Quite an interesting talk. I liked the format: "How to get more rps?..." It's humbling to think of the scale that some other folks deal with. I'm concerned of 2x growth with running a couple thousand sites on a dozen servers. You can't just hit up SO when you run into a problem at Facebook's scale.

We didn't get serious about load balancing and create a team dedicated to it until fall of 2011. At the time, Facebook had 800 million active users per month. We had run out of runway on commercial and open source options, and needed a lot more flexibility with how we allocated and moved traffic around. It took a lot of effort to get to where we are today, but the thing I'll tell you is that these huge systems always start with a prototype running on a dev server. And then you get it to run traffic for your whole team, and then the whole company. Then you try it in a cluster, a datacenter, a region, and all of a sudden you've built a load balancer.

It took Proxygen less than 18 months to go from inception at a hackathon to running all of Facebook's production HTTP traffic. And we could have shaved that by a few months if we had made smarter decisions along the way.

Post reply on HN