Espresso – Google’s peering edge architecture • Hacker News Board

Re: Espresso – Google’s peering edge architecture

#101

Post by collinmanderson » Thu, Apr 06, 2017, 2:08 PM UTC

pas wrote: post #94

danpalmer wrote: post #80

Earlier quoted context omitted.

For clarity, do you mean that Google can, for example, run to 99% saturation all the time, whereas a typical ISP might have 30-40% average, with peaks to full saturation that causes high latency/packet loss when it occurs?

Yes, that's about right. Since they control both sides of the link, they can manage the flow from higher up on the [software] stack. Basically, if the link is getting saturated, the distributed system simply throttles some requests upstream by diverting traffic from places that result in traffic over that link. (And of course this requires a very complex control plane, but doable, and with proper [secondary] controls…

So I wonder if that means they can do TCP control flow without dropping packets.

Re: Espresso – Google’s peering edge architecture

#102

Post by collinmanderson » Thu, Apr 06, 2017, 3:02 PM UTC

hueving wrote: post #84

These presentations from Google are pretty irritating at these conferences. If you're familiar with the SDN field (as most ONS attendees would be), this presentation is essentially nothing but bragging about the scale at which they operate. There is no useful information in here to advance the state of the art, no new ideas, no publicly available implementations (closed or open source). It's just a very high-level ar…

I don't know much about SDN, but Google did seem to imply that DNS responses based on querier is not a very good solution.

> Rather than pick a static point to connect users simply based on their IP address (or worse, the IP address of their DNS resolver), we dynamically choose the best point and rebalance our traffic based on actual performance data. Similarly, we are able to react in real-time to failures and congestion both within our network and in the public Internet.

Re: Espresso – Google’s peering edge architecture

#103

Post by SaveTheRbtz » Mon, Apr 10, 2017, 9:26 AM UTC

apanda wrote: post #39

This vision seems very similar to the 2011 talk by Scott Shenker: https://www.youtube.com/watch?v=YHeyuD89n1Y

This is a good talk about decomposability of control plane and creating proper abstractions for it.

PS. "The ability to master complexity is not the same as the ability to extract simplicity" is a good takeaway. PPS. This is a part of EE 122: https://inst.eecs.berkeley.edu/~ee122/fa12/class.html PPPS. PDF for SDN lecture: https://inst.eecs.berkeley.edu/~ee122/fa12/notes/25-SDN.pdf

Re: Espresso – Google’s peering edge architecture

#104

Post by pas » Tue, Apr 11, 2017, 9:43 AM UTC

joshbaptiste wrote: post #16

dkhenry wrote: post #7

I think with platforms like this it is now safe to say that the systems and services Google is deploying are no longer in the same category as classical networked systems. This is as foreign a concept from traditional networking and the seven layer OSI model as non von Neumann computing is from von neumann computing

Yep.. the CEO of Fastly has some cool presentations on YouTube of their SDN configuration which does away with the traditional router/switch combination for the switch/SDN server combo, which provides a much smarter network at a fraction of the cost. The only caveat is your engineers need to understand and tube the SDN software stack from the network card to the kernel.

Even though this reply is 6 days late, I finally had time to watch the mentioned talk, and it's interesting and still very relevant (it's from 2015):

https://www.youtube.com/watch?v=TLbzvbfWmfY interesting stuff starts around 7 minutes

Re: Espresso – Google’s peering edge architecture

#105

Post by pas » Tue, Apr 11, 2017, 9:48 AM UTC

collinmanderson wrote: post #101

pas wrote: post #94

Earlier quoted context omitted.

Yes, that's about right. Since they control both sides of the link, they can manage the flow from higher up on the [software] stack. Basically, if the link is getting saturated, the distributed system simply throttles some requests upstream by diverting traffic from places that result in traffic over that link. (And of course this requires a very complex control plane, but doable, and with proper [secondary] controls…

So I wonder if that means they can do TCP control flow without dropping packets.

I guess they do drop packets (it's the best - easiest/cheapest/cleanest - way to backpropagate pressure - aka backpressure), but they watch for it a lot more vigorously. Also as I understand they try to separate long lived connections (between DCs) from internal short lived traffic. Different teams, different patterns, different control structures.