"Google has one of the largest peering surfaces in the world, exchanging data with Internet Service Providers (ISPs) at 70 metros and generating more than 25 percent of all Internet traffic. " Wow.
70 metros?
[deleted]
Espresso – Google’s peering edge architecture
51–60 of 105 posts
Re: Espresso – Google’s peering edge architecture
#52Earlier quoted context omitted.
I'm sure he means something to do with caches; I had it on the tip of my tongue a moment ago, but the doorbell rang.
"There are only two hard problems in CS : 1. Naming things. 2. Cache invalidation. "
Only you got the order wrong. One of those distributed systems problems.
Re: Espresso – Google’s peering edge architecture
#53"Google has one of the largest peering surfaces in the world, exchanging data with Internet Service Providers (ISPs) at 70 metros and generating more than 25 percent of all Internet traffic. " Wow.
70 metros?
I wonder if they are referring to a peering Internet exchange point (IXP) when they say metro. Basically a building where networks converge and ISPs connect to each other.
Re: Espresso – Google’s peering edge architecture
#54"Google has one of the largest peering surfaces in the world, exchanging data with Internet Service Providers (ISPs) at 70 metros and generating more than 25 percent of all Internet traffic. " Wow.
70 metros?
70 metro areas. In a given metro area there might be one major traffic exchange point and de-facto most important peering location (example: SeattleIX in Seattle, and the Westin), or in a larger metro area, multiple exchange points.
edit: for a list of the geographical (OSI layer 1/2) locations where AS15169/google peers, see the following: https://www.peeringdb.com/asn/15169
Re: Espresso – Google’s peering edge architecture
#55Earlier quoted context omitted.
The google network is gold plated, lacks the jitter inherent in the internet at large or inside other competitors' networks. It makes it tempting to ignore some aspects of distributed computing, if only for a moment.
Could you expand a bit more on you comment? I feel I'm missing some context. Specifically, what do you mean by gold plated? Why is it tempting to ignore some aspects of distributed computing? I'm missing a lot of context that you are implicitly implying so could you elaborate?
It's gold plated because they basically built their own ISP by acquiring either:
a: dark fiber IRUs between cities/metro areas
b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo
c: some combination of A and B
and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and operates its own transport network around the US 48 states and around the world.
the exact design of what they're doing within their own AS at layers 1 and 2 is pretty opaque unless you happen to be a carrier partner that is willing to violate a whole raft of NDAs. But basically they've built their own backbone to a very massive scale yet without the huge capital expense of actually laying their own fiber between cities.
their network has incredibly low jitter because they don't run their links to saturation, and know EXACTLY what the latency is supposed to be from router interface to router interface between the pairs of core routers that are installed in each major city. Down to five decimal places, most likely. When you have your own dark fiber IRUs and operate your own WDM transport platforms you are in possession of things like OTDR traces for your dark fiber that tells you down to four decimal places the km length of your fiber path.
It also helps that the sort of people who have 'enable' on the AS15169 routers and core network gear are recruited from the top tier of network engineers and appropriately compensated. If they weren't working for Google they would be working for another major global player like NTT, DT, France Telecom/Orange, SingTel or Softbank.
Re: Espresso – Google’s peering edge architecture
#56Earlier quoted context omitted.
70 metros?
I wonder if they are referring to a peering Internet exchange point (IXP) when they say metro. Basically a building where networks converge and ISPs connect to each other.
yes, though "metro" is a better way to define it since many IXes are geographically distributed throughout their city. For example DE-CIX in frankfurt is in many different datacenters, with their core switches connected by DE-CIX controlled dark fiber. AMS-IX in amsterdam is in many facilities in the same metro area, all the same L2 peering fabric. The SIX in Seattle is in three facilities in the same metro and several local ISPs have built their own extensions of it to Vancouver BC.
Re: Espresso – Google’s peering edge architecture
#57I think with platforms like this it is now safe to say that the systems and services Google is deploying are no longer in the same category as classical networked systems. This is as foreign a concept from traditional networking and the seven layer OSI model as non von Neumann computing is from von neumann computing
> This is as foreign a concept from traditional networking and the seven layer OSI model as non von Neumann computing is from von neumann computing Not really. The OSI model doesn't say anything about where I run my routing algorithm and BGP application vs. where my actual switches are. "Classical" networking is an artifact of viewing routers/switches as monolithic blocks that embed all of their functionality in one…
> The OSI model doesn't say anything about where I run my routing algorithm and BGP application vs. where my actual switches are.
If you're $BIGASN and you set up an intra building singlemode crossconnect at $BIGCITY to establish settlement free peering (let's say for example a 4 x 10 Gbps bonded 802.3ad circuit) with $OTHERBIGASN, they most assuredly are going to notice if your BGP session and router is not directly on the other end of that cable.
Because they are going to be expecting sub-1ms latency to your router, and not "we're taking this session and stuffing it in some sort of tunnel or encapsulation and sending it somewhere else, to where the thing that actually speaks BGP is located". It's bad juju to practice deceptive peering.
Re: Espresso – Google’s peering edge architecture
#58Yup, this is the kind of thing you get when you put in $30B into infrastructure.
Re: Espresso – Google’s peering edge architecture
#59Earlier quoted context omitted.
Could you expand a bit more on you comment? I feel I'm missing some context. Specifically, what do you mean by gold plated? Why is it tempting to ignore some aspects of distributed computing? I'm missing a lot of context that you are implicitly implying so could you elaborate?
It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and op…
Where do you get the crazy idea that Google doesn't run its links to saturation? It's crazy because it would cost an enormous amount of money.
The B4 paper states multiple times that Google runs links at almost 100% saturation, versus the standard 30-40%. That's accomplished through the use of SDN technology and, even before that, through strict application of QoS.
https://web.stanford.edu/class/cs244/papers/b4-sigcomm2013.p...
A few more details about strategies here:
https://research.google.com/pubs/archive/45385.pdf
Then there's a whole bunch of other host-side optimizations, including the use of new congestion control algorithms.
http://queue.acm.org/detail.cfm?id=3022184
You might recognize the name of the last author...
Re: Espresso – Google’s peering edge architecture
#60Earlier quoted context omitted.
Could you expand a bit more on you comment? I feel I'm missing some context. Specifically, what do you mean by gold plated? Why is it tempting to ignore some aspects of distributed computing? I'm missing a lot of context that you are implicitly implying so could you elaborate?
It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and op…
@puzzle: while you're not wrong, do note that B4 is not (and is not designed to be) a low-latency, low-jitter network. It's designed for massive bandwidth for inter-datacenter data transfer.