The win in operating costs is impressive, bordering in the unbelievable (27x). Does anyone have a clue about where the win comes from?
Flat Datacenter Networks at Scale at Amazon
11–20 of 28 posts
Re: Flat Datacenter Networks at Scale at Amazon
#12One interesting consequence of this is that it's now (for the first time!) possible to get unlimited AWS egress. It's not cheap, and it's limited to `us-east-1`, but it's at least _possible_ now via AWS Interconnect: https://aws.amazon.com/interconnect/lastmile/pricing/
I don't think these things have anything to do with each other.
Re: Flat Datacenter Networks at Scale at Amazon
#13The win in operating costs is impressive, bordering in the unbelievable (27x). Does anyone have a clue about where the win comes from?
I am fairly confident that is a typo. The paper says "Our analysis reveals that RNG topologies are 9–45% cheaper than fat trees with equivalent oversubscription ratio." The blogger probably left off the % sign.
Re: Flat Datacenter Networks at Scale at Amazon
#14The win in operating costs is impressive, bordering in the unbelievable (27x). Does anyone have a clue about where the win comes from?
It's almost certainly meant to be 27%, not a factor of 27.
Re: Flat Datacenter Networks at Scale at Amazon
#15Good 6 minute video explainer: https://youtube.com/watch?v=yDoRYRRPOA0
I am moderately insulted that they want me to believe that graph theoreticians at Amazon sit at a desk with a bunch of optical T&M equipment. Sort of silly!
Re: Flat Datacenter Networks at Scale at Amazon
#16I get the feeling I am missing some info (Like what is meant here by Data Center Networks) thats preventing me from understanding whats happening here. I am guessing that this falls outside of the traditional rack/colo paradigm and has more to do with hyperscalers.
Re: Flat Datacenter Networks at Scale at Amazon
#17I get the feeling I am missing some info (Like what is meant here by Data Center Networks) thats preventing me from understanding whats happening here. I am guessing that this falls outside of the traditional rack/colo paradigm and has more to do with hyperscalers.
AWS hasn't published all that much about their network, but we know they use a high-radix folded Clos fabric because they mention it in their paper about Scalable Reliable Datagram. If you want an overview of the folded Clos fabric, try reading Google's "Jupiter Rising" paper. https://dl.acm.org/doi/pdf/10.1145/2829988.2787508
Re: Flat Datacenter Networks at Scale at Amazon
#18I get the feeling I am missing some info (Like what is meant here by Data Center Networks) thats preventing me from understanding whats happening here. I am guessing that this falls outside of the traditional rack/colo paradigm and has more to do with hyperscalers.
It's much larger but fundamentally it's not that different. In each rack you have one or two switches. How do you connect those racks to each other? The standard answer (simplified) is centralized spine switches but AWS discovered that a random network where the rack switches connect directly to each other is cheaper.
Re: Flat Datacenter Networks at Scale at Amazon
#19I get the feeling I am missing some info (Like what is meant here by Data Center Networks) thats preventing me from understanding whats happening here. I am guessing that this falls outside of the traditional rack/colo paradigm and has more to do with hyperscalers.
AWS hasn't published all that much about their network, but we know they use a high-radix folded Clos fabric because they mention it in their paper about Scalable Reliable Datagram. If you want an overview of the folded Clos fabric, try reading Google's "Jupiter Rising" paper. https://dl.acm.org/doi/pdf/10.1145/2829988.2787508
Then, if you want to know about using optical switches to connect Clos segments without a fixed spine, check out Google's "Mission Apollo" paper: https://arxiv.org/pdf/2208.10041
Re: Flat Datacenter Networks at Scale at Amazon
#20How is this different from Jellyfish? I recall reading a paper around that back in the 2010s
https://www.usenix.org/system/files/conference/nsdi12/nsdi12...
Edit: Answering my own question, Jellyfish proved theoretically that random networks can be better, and this is a working implementation based on that solves the problems with creating/operating random networks.