Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Billion User Load Balancer [video]

usenix.org

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Building a Billion User Load Balancer [video]

#31

Post by patrickshuff »

ju-st wrote: post #14

The "cartographer" doesn't use end user related information to select the best PoP? Or does "Sonar" measure the latency/throughput by looking at existing connections to users? Ok I watched the presentation now, Sonar apparently does that indeed.

Sonar is the system we use to measure latency from the client devices to all of our PoPs.

Cartographer is the system that consumes all of these sonar measurements and uses several other real time data sources (BGP routes, Link capacity, PoP health, PoP capacity, etc) and continually generates a GLB map for the most optimal targetting of requests to our PoPs.

Re: Building a Billion User Load Balancer [video]

#32

Post by degenerate »

nocarrier wrote: post #29
Retr0spectrum wrote: post #17

Earlier quoted context omitted.

I'm surprised that neither have any datacenters in India. Does anyone know why this might be?

Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries. Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk w…

Wow. Thank you for not bending over on the private key issue. That's terrible.

Re: Building a Billion User Load Balancer [video]

#33

Post by [deleted] »

nocarrier wrote: post #29
Retr0spectrum wrote: post #17

Earlier quoted context omitted.

I'm surprised that neither have any datacenters in India. Does anyone know why this might be?

Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries. Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk w…

[deleted]

Re: Building a Billion User Load Balancer [video]

#35

Post by sambe »

ape4 wrote: post #30
dorianm wrote: post #2

Map of Google data centers: http://imgur.com/l1dDdQe Map of Facebook data centers and PoP: http://imgur.com/dek8ESX

Too bad for Africa and Australia.

Yeah, massive surprise for me to see Australia/NZ not have anything dedicated. Network reliability over the whole region must be pretty poor and although in population terms it's probably too low, you'd think in purchasing power terms it would still be worth it.

Re: Building a Billion User Load Balancer [video]

#37

Post by nocarrier »

je42 wrote: post #34

wondering the tinydns and cartographer work together ? i didn't see any dynamic response capability in tinydns ? did i miss it in the docs of tinydns ?

Cartographer ingests BGP topology and POP and datacenter health (and other things), and then pushes what is basicially a lookup table to tinydns. None of the dynamism is inside the DNS server itself, which lets it focus on what it is good at--responding to a crap ton of DNS requests per second.

Re: Building a Billion User Load Balancer [video]

#38

Post by woodcut »

79d697i6fdif wrote: post #23
noponpop wrote: post #16

Interesting that Linux kernel performance (ipvs) is acceptable at l4 vs something like dpdk. I guess you just overcome the limitation by increasing the number of l4 instances load balanced by ecmp. Fun to see DSR in use. Also interesting to see that all the inherent problems with geolocation via gslb (DNS client IP is not the same as the real client IP) don't wind up being a big problem apparently. This seems to be a…

It's mostly because the point of DPDK and similar is to go around a lot of the processing in kernel, and IPVS does exactly this. I'm surprised IPVS isn't more popular, it's built into the kernel and extremely fast. HTTP proxy type load balancers are slugs in comparison Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this: 1) Balance requests using D…

What if you're not dealing with millions of connections but instead only a few thousand from whitelisted IP's and you need to optimise for high availability & latency? Could it be done with just anycast -> IPVS layer -> app servers ?

Re: Building a Billion User Load Balancer [video]

#39

Post by theDoug »

Retr0spectrum wrote: post #17
dorianm wrote: post #2

Map of Google data centers: http://imgur.com/l1dDdQe Map of Facebook data centers and PoP: http://imgur.com/dek8ESX

I'm surprised that neither have any datacenters in India. Does anyone know why this might be?

Google has announced a region for Mumbai (India) coming this year: https://cloud.google.com/about/locations/

(Can't/won't speculate as to why not earlier.)

Re: Building a Billion User Load Balancer [video]

#40

Post by justinsaccount »

noponpop wrote: post #16

Interesting that Linux kernel performance (ipvs) is acceptable at l4 vs something like dpdk. I guess you just overcome the limitation by increasing the number of l4 instances load balanced by ecmp. Fun to see DSR in use. Also interesting to see that all the inherent problems with geolocation via gslb (DNS client IP is not the same as the real client IP) don't wind up being a big problem apparently. This seems to be a…

> DNS client IP is not the same as the real client IP

see https://developers.google.com/speed/public-dns/faq

"I've read claims that Google Public DNS can slow down certain multimedia applications or websites. Are these true?

...

To help reduce the distance between DNS servers and users, Google Public DNS has deployed its servers all over the world. In particular, users in Europe should be directed to CDN content servers in Europe, users in Asia should be directed to CDN servers in Asia, and users in the eastern, central and western U.S. should be directed to CDN servers in those respective regions. We have also published this information to help CDNs provide good DNS results for multimedia users.

In addition, Google Public DNS engineers have proposed a technical solution called EDNS Client Subnet. This proposal allows resolvers to pass in part of the client's IP address (the first 24/64 bits or less for IPv4/IPv6 respectively) as the source IP in the DNS message, so that name servers can return optimized results based on the user's location rather than that of the resolver. To date, we have deployed an implementation of the proposal for many large CDNs (including Akamai) and Google properties. The majority of geo-sensitive domain names are already covered.

"

Post reply on HN