Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Espresso – Google’s peering edge architecture

blog.google

Post reply on HN
Print view View original

61–70 of 105 posts

Re: Espresso – Google’s peering edge architecture

#61

Post by puzzle »

walrus01 wrote: post #54
fiatjaf wrote: post #50

Earlier quoted context omitted.

70 metros?

70 metro areas. In a given metro area there might be one major traffic exchange point and de-facto most important peering location (example: SeattleIX in Seattle, and the Westin), or in a larger metro area, multiple exchange points. edit: for a list of the geographical (OSI layer 1/2) locations where AS15169/google peers, see the following: https://www.peeringdb.com/asn/15169

70 metros translate to many more POPs (also for redundancy reasons).

The metros can be seen in e.g. the 1e100.net hosts in a traceroute. They're usually the closest airport code, so e.g. lhr for London.

Somebody in China reverse engineered the metro/POP naming and addressing for latency reasons. You can see that, for example, there are at least three POPs in Sydney:

https://docs.google.com/spreadsheets/d/1a5HI0lkc1TycJdwJnCVD...

https://github.com/lennylxx/ipv6-hosts

Re: Espresso – Google’s peering edge architecture

#64

Post by smueller1234 »

nickpsecurity wrote: post #40
walrus01 wrote: post #23

Earlier quoted context omitted.

> This is as foreign a concept from traditional networking and the seven layer OSI model cough cough what? One of the major challenges for a CDN is predicated upon OSI layers 1 and 2: You need to establish POPs with routers and caching servers geographically distributed near major IX points (L2 peering fabrics, and crucial buildings that host the same IX points, where you can run intra-building fiber crossconnects fo…

In other words, it's built on hundreds of millions to billions of dollars of typical, boring stuff on the bottom that allows all the cool stuff to work as well as it does.

It's really easily billions, not hundreds of millions. I work for a much smaller outfit and we (handwave) push the smaller number already.

Re: Espresso – Google’s peering edge architecture

#65

Post by walrus01 »

puzzle wrote: post #59
walrus01 wrote: post #55

Earlier quoted context omitted.

It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and op…

Where do you get the crazy idea that Google doesn't run its links to saturation? It's crazy because it would cost an enormous amount of money. The B4 paper states multiple times that Google runs links at almost 100% saturation, versus the standard 30-40%. That's accomplished through the use of SDN technology and, even before that, through strict application of QoS. https://web.stanford.edu/class/cs244/papers/b4-sigco…

What I mean is that they do not run their links to saturation in the same way as an ordinary ISP. And because their traffic patterns are very different than an ordinary ISP, and much, much more geographically distributed, they can do all sorts of fun software tricks. The end result is the same: Low/no jitter and no packet loss.

As contrasted with what would happen if you had a theoretical hosting operation behind 2 x 10 Gbps transit connections to two upstreams, and tried to run both circuits at 8 to 9 Gbps outbound 24x7.

Re: Espresso – Google’s peering edge architecture

#66

Post by walrus01 »

puzzle wrote: post #61
walrus01 wrote: post #54

Earlier quoted context omitted.

70 metro areas. In a given metro area there might be one major traffic exchange point and de-facto most important peering location (example: SeattleIX in Seattle, and the Westin), or in a larger metro area, multiple exchange points. edit: for a list of the geographical (OSI layer 1/2) locations where AS15169/google peers, see the following: https://www.peeringdb.com/asn/15169

70 metros translate to many more POPs (also for redundancy reasons). The metros can be seen in e.g. the 1e100.net hosts in a traceroute. They're usually the closest airport code, so e.g. lhr for London. Somebody in China reverse engineered the metro/POP naming and addressing for latency reasons. You can see that, for example, there are at least three POPs in Sydney: https://docs.google.com/spreadsheets/d/1a5HI0lkc1Ty…

yes, the practice of using IATA airport codes or similar for reverse DNS is a practice that long pre-dates google. They just adopted industry best practices when they started building consistent rDNS.

Re: Espresso – Google’s peering edge architecture

#67

Post by walrus01 »

packetslave wrote: post #60
walrus01 wrote: post #55

Earlier quoted context omitted.

It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and op…

@puzzle: while you're not wrong, do note that B4 is not (and is not designed to be) a low-latency, low-jitter network. It's designed for massive bandwidth for inter-datacenter data transfer.

running your own internal links to near saturation (such as a theoretical 100 Gbps DWDM or MPLS circuit between two google datacenters in two different states) is a very different thing than running a BGP edge connection to saturation, such as a theoretical 100 Gbps, short reach intra building crossconnect from a huge CDN such as Limelight to a content-sink ISP such as Charter/TWTC or Comcast.

Re: Espresso – Google’s peering edge architecture

#68

Post by biokoda »

piyushpr134 wrote: post #63

One biggest takeaway from this is that they can have multiple machines for the same IP address. That is just awesome and also explains how they have probably managed to scale up services 8.8.8.8 without needing to use load balancers.

Anycast is pretty standard.

Re: Espresso – Google’s peering edge architecture

#69

Post by mmarx »

biokoda wrote: post #68
piyushpr134 wrote: post #63

One biggest takeaway from this is that they can have multiple machines for the same IP address. That is just awesome and also explains how they have probably managed to scale up services 8.8.8.8 without needing to use load balancers.

Anycast is pretty standard.

Especially so for DNS servers, and since long before the google nameservers[0].

[0] https://tools.ietf.org/html/rfc3258

Post reply on HN