Live data from Hacker News

Speed, scale and reliability: 25 years of Google datacenter networking evolution

cloud.google.com

41–50 of 91 posts

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#41

Earlier quoted context omitted.

[flagged]

Yea, you're not the only "insider" here. And you're 100% wrong. Just because you completely misunderstand what those Amazon/MS employees are doing in those buildings doesn't mean that you know what you're talking about. The big cloud players have the vast majority of their compute and storage hosted out of their own custom built and self-owned data centers. The stuff you see in colos is just the edge locations like C…

[flagged]

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#42

Earlier quoted context omitted.

Yea, you're not the only "insider" here. And you're 100% wrong. Just because you completely misunderstand what those Amazon/MS employees are doing in those buildings doesn't mean that you know what you're talking about. The big cloud players have the vast majority of their compute and storage hosted out of their own custom built and self-owned data centers. The stuff you see in colos is just the edge locations like C…

[flagged]

Well those people lied to you then, or more likely there was a misunderstanding, because you can literally just look up the sites I mentioned above and see that you're entirely incorrect.

You don't need to be under NDA to see the hundreds of billions of dollars worth of custom built and self-owned data centers that the big players have.

Hell, you can literally just look at their public websites: https://www.google.com/about/datacenters/locations/

I am one of those "pay grades many layers higher", and I can personally confirm that each of the locations above is wholly owned and used by Google, and only Google, which already invalidates your claim that "you can count the wholly-owned sites on one hand". Again, this isn't secret info, so I have no issue sharing it.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#43

It seems all cutting edge datacenters like x.ai Colossus are using Nvidia networking. Now Google is upgrading to Nvidia networking, too. Since Nvidia owns most of the Gpgpu products, they have top notch networking and interconnect, I wonder if they don't have a plan to own all datacenter hardware in the future. Maybe they plan to also release CPUs, motherboards, storage and whatever else is needed.

I read this slightly differently, that specific machine types with Nvidia GPU hardware also have Nvidia networking for tying together those GPUs. Google has its own TPUs and don’t really use GPUs except to sell them to end customers on cloud I think. So using Nvidia networking for Nvidia GPUs across many machines on cloud is really just a reflection of what external customers want to buy. Disclaimer, I work at Google…

Having just worked with some of the Thread folks at M&S, thought I'd reach out and say hello. Seems like it was an awesome team! (=

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#44
post #25

Awesome Google... Now learn what an availability zone is and stop creating them with firewalls across the same data center. Oh and make your data centers smaller. Not so big they can be seen in Google Maps. Because otherwise, you will be unable to move those whale sized workloads to an alternative. https://youtu.be/mDNHK-SzXEM?t=564 https://news.ycombinator.com/item?id=35713001 "Unmasking Google Cloud: How to Determi…

Making a datacenter not visible from Google Maps, at least on most big cities where Google zones are deployed, would mean making them smaller than a car. Or even smaller than a dishwasher. If I check London (where europe-west2 is kinda located) on Google Maps right now, I can easily discern manhole covers or people. If I check Jakarta (Asia-southeast2) things smaller than a car get confusing, but you can definitely s…

Your comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#45

Earlier quoted context omitted.

[flagged]

Well those people lied to you then, or more likely there was a misunderstanding, because you can literally just look up the sites I mentioned above and see that you're entirely incorrect. You don't need to be under NDA to see the hundreds of billions of dollars worth of custom built and self-owned data centers that the big players have. Hell, you can literally just look at their public websites: https://www.google.co…

[flagged]

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#46
post #17

This mentions Jupiter generations, which I think is about 10-15 years old at this point. It doesn't really talk about what existed before so it's not really 25 years of history here. I want to say "Watchtower" was before Jupiter? but honestly it's been about a decade since I read anything about it. Google's DC networking is interesting because of how deeply integrated it is into the entire software stack. Click on so…

I would guess the Nvidia ConnectX is part of a secondary networking plane, not plugged into Jupiter. Current-gen Google NICs are custom hardware with a _lot_ of Google-specific functionality, such as running the borglet on the NIC to free up all CPU cores for guests.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#47

Earlier quoted context omitted.

Well those people lied to you then, or more likely there was a misunderstanding, because you can literally just look up the sites I mentioned above and see that you're entirely incorrect. You don't need to be under NDA to see the hundreds of billions of dollars worth of custom built and self-owned data centers that the big players have. Hell, you can literally just look at their public websites: https://www.google.co…

[flagged]

I'm not trying to make you divulge anything. I don't particularly care who you talk to, or who you are, nor do I care if you take it as a "personal insult" that you might be wrong.

You are right that it would be nuts that multiple senior people would collude to lie to you, which is why it's almost certainly more likely that you are just misunderstanding the information that was provided to you. It's possible to prove that you are incorrect based on publicly available data from multiple different sources. You can keep being stubborn if you want, but that won't make any of your statements correct.

You didn't ask for my advice, but I'll give it anyway: try to be more open to the possibility that you're wrong, especially when evidence that you're wrong is right in front of you. End of story.

Re: Speed, scale and reliability: 25 years of Google datacenter networking evolution

#50

Earlier quoted context omitted.

This isn't even close to true. You can just go on Google Maps and visually see the literally *hundreds* of wholly-owned and custom built data centers from AWS, MS, and Google. Edge locations (like Cloud CDN) are often in colos, but the main regions compute/storage are not. Most of them are even labeled on Google Maps. Here's a couple search terms you can just type into Google Maps and see a small fraction of what I m…

[flagged]

It is true that every cloud provider uses some edge/colo infra, but it is also not true that most (or even really any relevant) processing happens in those colo/edge locations.

Google lists their dc locations publicly: https://www.google.com/about/datacenters/locations/

Aws doesn't list the campuses as publicly, but https://aws.amazon.com/about-aws/global-infrastructure/regio... shows the AZ vs edge deployments and any situation with multiple AZs is going to have buildings, not floors operated by Amazon.

And limiting to just outside the US, both aws and Google have more than ten wholly owned campuses each, and then on top of that, there is edge/colo space.

Post reply on HN