Then Groq should reign emperor?
TPUs vs. GPUs and why Google is positioned to win AI race in the long term
161–170 of 328 posts
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#162Earlier quoted context omitted.
Google Hangouts wasn't small. Google+ was big and supposedly "the future" and is the canonical example of a huge misallocation of resources. Google will have no problem discontinuing Google "AI" if they finally notice that people want a computer to shut up rather than talk at them.
> Google+ was big how you define big? My understanding they failed to compete with facebook, and decided to redirect resources somewhere else.
At the time Google+ was started and shortly after, leadership (larry page at that time) focused the attention of the company on it. There was a social bonus (that you'd get if you integrated your product), there were large changes to existing systems to support Google+, and the company made it quite clear it thought that social was the direction to go and that Google+ was going to be an enormous product.
I and a lot of other googlers were really confused by all of this because at the time we were advocating that Google put more effort into its nascent cloud business (often to get the reply "but we already have appengine" or "cloud isn't as profitable as ads") and that social, while getting a lot of attention, wasn't really a good business for google to be in (with a few exceptions like Orkut and Youtube, Google's attempts at social have been pretty uninspired).
There were even books written at the time that said Google looked lazy and slow and that Meta was going to eat their lunch. But shortly after Google+ tanked, Google really began to focus on Cloud (in a way that pissed off a lot of Googlers in the same way Google+ did- by taking resources and attention from other projects). Now, Meta looks like its going to have a challenging future while Google is on to achieving what Larry Page originally intended: a reliable revenue stream that is reinvested into development of true AI.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#163Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects. To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale…
It's fun when then you read last Nvidia tweet [1] suggesting that still their tech is better, based on pure vibes as anything in the (Gen)AI-era. [1] https://x.com/nvidianewsroom/status/1993364210948936055
Not vibes. TPUs have fallen behind or had to be redesigned from scratch many times as neural architectures and workloads evolved, whereas the more general purpose GPUs kept on trucking and building on their prior investments. There's a good reason so much research is done on Nvidia clusters and not TPU clusters. TPU has often turned out to be over-specialized and Nvidia are pointing that out.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#164> It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prim…
CUDA is just a better dev experience. Lots of training is experiments where developer/researcher productivity matters. Googlers get to use what they're given, others get to choose.
Once you settle on a design then doing ASICs to accelerate it might make sense. But I'm not sure the gap is so big, the article says some things that aren't really true of datacenter GPUs (Nvidia dc gpus haven't wasted hardware on graphics related stuff for years).
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#165They can only privatize the AI race.
If Google wins, we all lose.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#166Earlier quoted context omitted.
It's fun when then you read last Nvidia tweet [1] suggesting that still their tech is better, based on pure vibes as anything in the (Gen)AI-era. [1] https://x.com/nvidianewsroom/status/1993364210948936055
Not vibes. TPUs have fallen behind or had to be redesigned from scratch many times as neural architectures and workloads evolved, whereas the more general purpose GPUs kept on trucking and building on their prior investments. There's a good reason so much research is done on Nvidia clusters and not TPU clusters. TPU has often turned out to be over-specialized and Nvidia are pointing that out.
You say that like I d a bad thing. Nvidia architectures keep changing and getting more advanced as well, with specialized tensor operations, different accumulators and caches, etc. I see no issue with progress.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#167Earlier quoted context omitted.
So when the bubble pops the companies making the shovels (TSMC, NVIDIA) might still have the money they got for their products and some of the ex-AI companies might least be able to sell standard compliant GPUs on the wider market. And Google will end up with lots of useless super specialized custom hardware.
How could Google's custom hardware become useless? They've used it for their business for years now and will do so for years into the future. It's not like their hardware is LLM specific. Google cannot lose with their vast infrastructure. Meanwhile OpenAI et al dumping GPUs while everyone else is doing the same will get pennies on the dollar. It's exactly the opposite to what you describe. I hope that comes to pass,…
Same way cloud hardware always risks becoming useless. The newer hardware is so much better you can't afford to not upgrade, e.g. an algorithmic improvement that can be run on CUDA devices but not on existing TPUs, which changes the economics of AI.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#168Earlier quoted context omitted.
For users buying H200s for AI workloads, the "ASIC" tensor cores deliver the overwhelming bulk of performance. So they already do this, and have been since Volta in 2017. To put it into perspective, the tensor cores deliver about 2,000 TFLOPs of FP8, and half that for FP16, and this is all tensor FMA/MAC (comprising the bulk of compute for AI workloads). The CUDA cores -- the rest of the GPU -- deliver more in the 70…
Is it the Cuda cores that run the vertex/fragment/etc shaders in normal GPUs? Where does the ray tracing units fit in? How much of a modern Nvidia GPU is general purpose vs specialized to graphics pipelines?
A datacenter GPU has next to nothing left related to graphics. You can't use them to render graphics. It's a pure computational kernel machine.
Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term
#169Earlier quoted context omitted.
Catch-up in what exactly? Google isn't building hardware to sell, they aren't in the same market. Also I feel you completely misunderstand that the problem isn't how fast is ONE gpu vs ONE tpu, what matters is the costs for the same output. If I can fill a datacenter at half the cost for the same output, does it matters I've used twice the TPUs and that a single Nvidia Blackwell was faster? No... And hardware cost is…
Also, isn't memory a bit moot? At scale I thought that the ASICs frequently sat idle waiting for memory.
You're doing operations on the memory once it's been transferred to gpu memory. Either shuffling it around various caches or processors or feeding it into tensor cores or other matrix operations. You don't want to be sitting idle.