Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed, scale and reliability: 25 years of Google datacenter networking evolution

cloud.google.com

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#51

Post by joshuamorton »

belter wrote: post #44
tecleandor wrote: post #26

Earlier quoted context omitted.

Making a datacenter not visible from Google Maps, at least on most big cities where Google zones are deployed, would mean making them smaller than a car. Or even smaller than a dishwasher. If I check London (where europe-west2 is kinda located) on Google Maps right now, I can easily discern manhole covers or people. If I check Jakarta (Asia-southeast2) things smaller than a car get confusing, but you can definitely s…

Your comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.

What if you have dozens of big data centers?

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#52

Post by toast0 »

traceroute66 wrote: post #39
anewplace wrote: post #38

Earlier quoted context omitted.

This isn't even close to true. You can just go on Google Maps and visually see the literally *hundreds* of wholly-owned and custom built data centers from AWS, MS, and Google. Edge locations (like Cloud CDN) are often in colos, but the main regions compute/storage are not. Most of them are even labeled on Google Maps. Here's a couple search terms you can just type into Google Maps and see a small fraction of what I m…

[flagged]

It can be true that all the big clouds/cdns/websites are in all the big colos and that big tech also has many owned and operated sites elsewhere.

As one of these big companies. You've got to be in the big colos because that's where you interconnect and peer. You don't want to have a full datacenter installation at one of these places if you can avoid it, because costs are high; but building your own has a long timetable, so it makes sense to put things into colos from time to time and of course, things get entrenched.

I've seen datacenter lists when I worked at Yahoo and Facebook, and it was a mix of small installations at PoPs, larger installations at commercial colo facilities, and owned and operated data centers. Usually new large installations were owned and operated, but it took a long time to move out of commercial colos too. And then there's also whole building leases, from companies that specialize in that. Outside the US, there was more likely hood of being in commercial colo, I think because of logistics, but at large system counts, the dollar efficiency of running it yourself becomes more appealing (assuming land, electricity, and fiber are available)

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#53

Post by pknomad »

anewplace wrote: post #42
traceroute66 wrote: post #41

Earlier quoted context omitted.

[flagged]

Well those people lied to you then, or more likely there was a misunderstanding, because you can literally just look up the sites I mentioned above and see that you're entirely incorrect. You don't need to be under NDA to see the hundreds of billions of dollars worth of custom built and self-owned data centers that the big players have. Hell, you can literally just look at their public websites: https://www.google.co…

Ignore him.

He's got a tinfoil hat on and won't be persuaded..

> Because I'm not relying on what one person or one company told me, my facts have been diligently and discretely cross-checked.

"Discretely cross-checked" already tells me he chooses to live in his own reality.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#54

Post by dekhn »

anewplace wrote: post #42
traceroute66 wrote: post #41

Earlier quoted context omitted.

[flagged]

Well those people lied to you then, or more likely there was a misunderstanding, because you can literally just look up the sites I mentioned above and see that you're entirely incorrect. You don't need to be under NDA to see the hundreds of billions of dollars worth of custom built and self-owned data centers that the big players have. Hell, you can literally just look at their public websites: https://www.google.co…

I assume they are referring to PoPs, or other locations where large providers house frontends and other smaller resources.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#55

Post by cavisne »

cletus wrote: post #17

This mentions Jupiter generations, which I think is about 10-15 years old at this point. It doesn't really talk about what existed before so it's not really 25 years of history here. I want to say "Watchtower" was before Jupiter? but honestly it's been about a decade since I read anything about it. Google's DC networking is interesting because of how deeply integrated it is into the entire software stack. Click on so…

The past few years there has been a weird situation where Google and AWS have had worse GPU's than smaller providers like Coreweave + Lambda Labs. This is because they didn't want to buy into Nvidias proprietary Infiniband stack for GPU-GPU networking, and instead wanted to make it work on top of their ethernet (but still pretty proprietary) stack.

The outcome was really bad GPU-GPU latency & bandwidth between machines. My understanding is ConnectX is Nvidias supported (and probably still very profitable) way for these hyperscalers to use their proprietary networks without buying Infiniband switches and without paying the latency cost of moving bytes from the GPU to the CPU.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#56

Post by latchkey »

cavisne wrote: post #55
cletus wrote: post #17

This mentions Jupiter generations, which I think is about 10-15 years old at this point. It doesn't really talk about what existed before so it's not really 25 years of history here. I want to say "Watchtower" was before Jupiter? but honestly it's been about a decade since I read anything about it. Google's DC networking is interesting because of how deeply integrated it is into the entire software stack. Click on so…

The past few years there has been a weird situation where Google and AWS have had worse GPU's than smaller providers like Coreweave + Lambda Labs. This is because they didn't want to buy into Nvidias proprietary Infiniband stack for GPU-GPU networking, and instead wanted to make it work on top of their ethernet (but still pretty proprietary) stack. The outcome was really bad GPU-GPU latency & bandwidth between machin…

Your understanding is correct. Part of the other issue is that at one point, there was a huge shortage of availability of IB switches... lead times of 1+ years... another solution had to be found.

RoCE is IB over Ethernet. All the underlying documentation and settings to put this stuff together are the same. It doesn't require ConnectX NIC's though. We do the same with 8x Broadcom Thor 2 NIC's (into a Broadcom Tomahawk 5 based Dell Z9864F switch) for our own 400G cluster.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#58

Post by jiggawatts »

joshuamorton wrote: post #51
belter wrote: post #44

Earlier quoted context omitted.

Your comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.

What if you have dozens of big data centers?

To reinforce your point:

The scale of cloud data centres reflects the scale of their customer base, not the size of the basket for each individual customer.

Larger data centres actually improve availability through several mechanisms: more power components such as generators means the failure of any one is just a few percent instead of a total blackout. You can also partition core infrastructure like routers and power rails into more fault domains and update domains.

Some large clouds have two update domains and five fault domains on top of three zones that are more than 10km apart. You can’t beat ~30 individual partitions with your data centres at a reasonable cost!

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#59

Post by danpalmer »

dmacedo wrote: post #43
danpalmer wrote: post #11

Earlier quoted context omitted.

I read this slightly differently, that specific machine types with Nvidia GPU hardware also have Nvidia networking for tying together those GPUs. Google has its own TPUs and don’t really use GPUs except to sell them to end customers on cloud I think. So using Nvidia networking for Nvidia GPUs across many machines on cloud is really just a reflection of what external customers want to buy. Disclaimer, I work at Google…

Having just worked with some of the Thread folks at M&S, thought I'd reach out and say hello. Seems like it was an awesome team! (=

You're lucky to be working with them, an amazing team.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#60

Post by dafugg »

immibis wrote: post #30
adrian_b wrote: post #13

Earlier quoted context omitted.

Nvidia networking is what used to be called Mellanox networking, which was already dominant in datacenters.

Only within supercomputers (including the smaller GPU ones used to train AI). Normal data centers use Cisco or Juniper or similarly.well known Ethernet equipment, and they still do. The Mellanox/Nvidia Infiniband networks are specifically used for supercomputer-like clusters.

You seem to have a narrow definition of “normal” for datacenters. Meta were using OCP mellanox NICs for common hardware platforms a decade ago and still are.
Post reply on HN