Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Billion User Load Balancer [video]

usenix.org

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Building a Billion User Load Balancer [video]

#21

Post by mafribe »

briandear wrote: post #15
brobinson wrote: post #5

Earlier quoted context omitted.

Both Google and Facebook both have datacenters/PoPs in Taiwan? I can't imagine the 24mm people there justify it, and both services are either largely unused or blocked in the nearby PRC... I'm more surprised that Google has no presence in Japan, though.

There's a major Asia-Pacific undersea cable connected to Taiwan, so Taiwan is a useful Asia-Pacific gateway.

More precisely, Taiwan is well connected by direct cables to Japan, China, Philippines, Viet Nam, Malaysia, Thailand, Korea, Singapore and the US. As far as I'm aware Google owns a cable connecting Taiwan with Japan (part of [1]).

[1] https://en.wikipedia.org/wiki/FASTER_(cable_system)

Re: Building a Billion User Load Balancer [video]

#23

Post by 79d697i6fdif »

noponpop wrote: post #16

Interesting that Linux kernel performance (ipvs) is acceptable at l4 vs something like dpdk. I guess you just overcome the limitation by increasing the number of l4 instances load balanced by ecmp. Fun to see DSR in use. Also interesting to see that all the inherent problems with geolocation via gslb (DNS client IP is not the same as the real client IP) don't wind up being a big problem apparently. This seems to be a…

It's mostly because the point of DPDK and similar is to go around a lot of the processing in kernel, and IPVS does exactly this. I'm surprised IPVS isn't more popular, it's built into the kernel and extremely fast.

HTTP proxy type load balancers are slugs in comparison

Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this:

1) Balance requests using DNS anycast so you can spread load before it hits your servers

2) Setup "Head End" machines with as large pipes as possible (40Gbps?) and load balance at the lowest layer you can. Balance at IP level using IPVS and direct server return. A single reasonable machine can handle a 40Gbps pipe. I guess you could setup a bunch of these but I doubt many people are over 40Gpbs. Oh, and don't use cloud services for these. The virtualization overhead is high on the network plane and even with SR-IOV you don't get access to all hardware NIC queues. Also, I don't know of any cloud provider thats compatible with direct server return since they typically virtualize your "private cloud" at layer 3, whereas IPVS actually touches layer 2 a little. Do yourself a favor and get yourself a few colo's for your load balancers.

3) Setup a ton of HTTP-proxy type load balancers. This includes Nginx, Varnish, Haproxy etc... One of these machines can probably handle 1-5 Gbps of traffic so expect 20 or so behind each layer 3 balancer. These NEED to be hardened substantially because most attacks will be layer 4 and up once an adversary realizes they can't just flood you out(due to powerful IPVS balancers above). SYN cookies are extremely important here since you're dealing with TCP... just try to set everything up to avoid storing TCP state at all costs. This also means no NAT. You might want to keep these in the colo with your L3 load balancers.

4) Now for your app servers. Depending on if you're using a dog slow language or not, you'll want between 3 and 300 app servers behind each HTTP proxy. You don't really need to harden these as much since the traffic is lower and any traffic that reaches here is clean HTTP. Go ahead and throw these on the cloud if want

Re: Building a Billion User Load Balancer [video]

#24

Post by true_religion »

brobinson wrote: post #5
dorianm wrote: post #2

Map of Google data centers: http://imgur.com/l1dDdQe Map of Facebook data centers and PoP: http://imgur.com/dek8ESX

Both Google and Facebook both have datacenters/PoPs in Taiwan? I can't imagine the 24mm people there justify it, and both services are either largely unused or blocked in the nearby PRC... I'm more surprised that Google has no presence in Japan, though.

In my experience, having a PoP in Taiwan, means that some data requests from S. Korea, Philippines, and Australia are routed to that.

I never know how that works, since Singapore and Tokyo, Japan should be closer---but it still happens.

Re: Building a Billion User Load Balancer [video]

#25

Post by pyvpx »

true_religion wrote: post #24
brobinson wrote: post #5

Earlier quoted context omitted.

Both Google and Facebook both have datacenters/PoPs in Taiwan? I can't imagine the 24mm people there justify it, and both services are either largely unused or blocked in the nearby PRC... I'm more surprised that Google has no presence in Japan, though.

In my experience, having a PoP in Taiwan, means that some data requests from S. Korea, Philippines, and Australia are routed to that. I never know how that works, since Singapore and Tokyo, Japan should be closer---but it still happens.

load and network conditions (paths from you to Tokyo are experiencing packet loss for $REASONS, Singapore is overloaded/broken/full, etc).

Re: Building a Billion User Load Balancer [video]

#26

Post by noponpop »

79d697i6fdif wrote: post #23
noponpop wrote: post #16

Interesting that Linux kernel performance (ipvs) is acceptable at l4 vs something like dpdk. I guess you just overcome the limitation by increasing the number of l4 instances load balanced by ecmp. Fun to see DSR in use. Also interesting to see that all the inherent problems with geolocation via gslb (DNS client IP is not the same as the real client IP) don't wind up being a big problem apparently. This seems to be a…

It's mostly because the point of DPDK and similar is to go around a lot of the processing in kernel, and IPVS does exactly this. I'm surprised IPVS isn't more popular, it's built into the kernel and extremely fast. HTTP proxy type load balancers are slugs in comparison Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this: 1) Balance requests using D…

I saw ipvs was implemented in kernel but didn't realize it bypassed the stack. Thanks for clarifying.

Re: Building a Billion User Load Balancer [video]

#27

Post by pyvpx »

bschwindHN wrote: post #8
brobinson wrote: post #5

Earlier quoted context omitted.

Both Google and Facebook both have datacenters/PoPs in Taiwan? I can't imagine the 24mm people there justify it, and both services are either largely unused or blocked in the nearby PRC... I'm more surprised that Google has no presence in Japan, though.

Google has an office in Tokyo but I guess not a datacenter.

but plenty of Google Global Caches, so performance (latency to cached objects and DNS queries) is still quite good

Re: Building a Billion User Load Balancer [video]

#28

Post by 79d697i6fdif »

noponpop wrote: post #26
79d697i6fdif wrote: post #23

Earlier quoted context omitted.

It's mostly because the point of DPDK and similar is to go around a lot of the processing in kernel, and IPVS does exactly this. I'm surprised IPVS isn't more popular, it's built into the kernel and extremely fast. HTTP proxy type load balancers are slugs in comparison Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this: 1) Balance requests using D…

I saw ipvs was implemented in kernel but didn't realize it bypassed the stack. Thanks for clarifying.

not all of it. IVPS runs between layer 2 and 3 if using direct server return. It does bypass quite a bit though...

Re: Building a Billion User Load Balancer [video]

#29

Post by nocarrier »

Retr0spectrum wrote: post #17
dorianm wrote: post #2

Map of Google data centers: http://imgur.com/l1dDdQe Map of Facebook data centers and PoP: http://imgur.com/dek8ESX

I'm surprised that neither have any datacenters in India. Does anyone know why this might be?

Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries.

Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk worth it. It could be an expensive paperweight if the Indian government changed their minds.

Post reply on HN