Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flat Datacenter Networks at Scale at Amazon

perspectives.mvdirona.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Flat Datacenter Networks at Scale at Amazon

#21

Post by trumpdong »

socketcluster wrote: post #5

Interesting reading this because this is essentially the principle behind https://socketcluster.io/ scalability; the sharding of channels across available brokers is pseudo-random. It uses a hash function for determinism but the distribution appears to be random and that was also the best way I could find to distribute load evenly between available nodes. It is key to its embarrassingly parallel design. It's interest…

That's a different thing entirely, that assumes you already have a physical layer that allows any client to connect to any broker, this is about building that physical layer

Re: Flat Datacenter Networks at Scale at Amazon

#22

Post by trumpdong »

kev009 wrote: post #4

It's not that dissimilar to how the Internet works. Although you have some steering like IX peering switches, and social/economic factors, but in whole it is fairly random.

On the internet each ISP is looking for congested links and expanding those links, like a lazy-loaded fat tree.

Re: Flat Datacenter Networks at Scale at Amazon

#23

Post by trumpdong »

cyberax wrote: post #9

One interesting consequence of this is that it's now (for the first time!) possible to get unlimited AWS egress. It's not cheap, and it's limited to `us-east-1`, but it's at least _possible_ now via AWS Interconnect: https://aws.amazon.com/interconnect/lastmile/pricing/

It's not a consequence of this.

And that's their... third... fourth? completely separate system for connecting an AWS virtual network to a physical DC in the outside world?

Re: Flat Datacenter Networks at Scale at Amazon

#24

Post by cyberax »

trumpdong wrote: post #23
cyberax wrote: post #9

One interesting consequence of this is that it's now (for the first time!) possible to get unlimited AWS egress. It's not cheap, and it's limited to `us-east-1`, but it's at least _possible_ now via AWS Interconnect: https://aws.amazon.com/interconnect/lastmile/pricing/

It's not a consequence of this. And that's their... third... fourth? completely separate system for connecting an AWS virtual network to a physical DC in the outside world?

Yes, there are several other options. This is the first one that is not metered.

Re: Flat Datacenter Networks at Scale at Amazon

#25

Post by cyberax »

wmf wrote: post #12
cyberax wrote: post #9

One interesting consequence of this is that it's now (for the first time!) possible to get unlimited AWS egress. It's not cheap, and it's limited to `us-east-1`, but it's at least _possible_ now via AWS Interconnect: https://aws.amazon.com/interconnect/lastmile/pricing/

I don't think these things have anything to do with each other.

I heard from several engineers several years ago that egress was metered because their network did not have enough switching capacity to make it truly unmetered at AWS scale. Looks like this might have been a part of the solution.

Re: Flat Datacenter Networks at Scale at Amazon

#26

Post by everfrustrated »

cyberax wrote: post #24
trumpdong wrote: post #23

Earlier quoted context omitted.

It's not a consequence of this. And that's their... third... fourth? completely separate system for connecting an AWS virtual network to a physical DC in the outside world?

Yes, there are several other options. This is the first one that is not metered.

Might not be GB metered, but it is metered by the port-size/Hour at a very heavily marked up rate.

Re: Flat Datacenter Networks at Scale at Amazon

#27

Post by alsetmusic »

I recognize this blog from the layout / design. It was impressive the previous time I say it. This is pretty far beyond my level of comprehension, but I get the gist of it and it's pretty interesting. I don't understand why this works better, though. It's sort of, "the math makes it this way" and I don't know why. I do understand how it supersedes trees, so that's pretty cool for future networking.

Re: Flat Datacenter Networks at Scale at Amazon

#28

Post by nicholasbraker »

There's some important nuances stated by Ivan Pepelnjak who is considered an authority in networking: https://blog.ipspace.net/2026/06/goodbye-leaf-spine-networks...

Although very interesting from a scientific point of view I always consider these use cases to be very unique to AWS / hyperscaler DC's and in no way a new 'blueprint' for how to build DC-networks/fabrics.

Post reply on HN