Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TPUs vs. GPUs and why Google is positioned to win AI race in the long term

uncoveralpha.com

Post reply on HN
Print view View original

171–180 of 328 posts

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#171

Post by heisenbit »

timmg wrote: post #115
m4rtink wrote: post #103

Earlier quoted context omitted.

So when the bubble pops the companies making the shovels (TSMC, NVIDIA) might still have the money they got for their products and some of the ex-AI companies might least be able to sell standard compliant GPUs on the wider market. And Google will end up with lots of useless super specialized custom hardware.

> And Google will end up with lots of useless super specialized custom hardware. If it gets to the point where this hardware is useless (I doubt it), yes Google will have it sitting there. But it will have cost Google less to build that hardware than any of the companies who built on Nvidia.

And running loads long term profitable may require both lower power use as well as longer chip lifetimes - something associated with lower power use.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#172

Post by jeffbee »

Traster wrote: post #132
1980phipsi wrote: post #60

> It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prim…

Training is taking an enormous problem and trying to break it into lots of pieces and managing the data dependency between those pieces. It's solving 1 really hard problem. Inference is the opposite, it's lots of small independent problems. All of this "we have X many widgets connected to Y many high bandwidth optical telescopes" is all a training problem that they need to solve. Inference is "I have 20 tokens and I…

I can't think of any case where inference doesn't care about latency.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#173

Post by thesz »

HarHarVeryFunny wrote: post #77
thesz wrote: post #58

5 days ago: https://news.ycombinator.com/item?id=45926371 Sparse models have same quality of results but have less coefficients to process, in case described in the link above sixteen (16) times as less. This means that these models need 8 times less data to store, can be 16 and more times faster and use 16+ times less energy. TPUs are not all that good in the case of sparse matrices. They can be used to train dense…

TPUs do include dedicated hardware, SparseCores, for sparse operations. https://docs.cloud.google.com/tpu/docs/system-architecture-t... https://openxla.org/xla/sparsecore

SparseCores appear to be block-sparse as opposed to element-sparse. They use 8- and 16-wide vectors to compute.

Here's another inference-efficient architecture where TPUs are useless: https://arxiv.org/pdf/2210.08277

There is no matrix-vector multiplication. Parameters are estimated using Gumbel-Softmax. TPUs are of no use here.

Inference is done bit-wise and most efficient inference is done after application of boolean logic simplification algorithms (ABC or mockturtle).

In my (not so) humble opinion, TPUs are example case of premature optimization.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#174

Post by sagarm »

timmg wrote: post #20
sbarre wrote: post #2

A question I don't see addressed in all these articles: what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.

They will, I'm sure. The big difference is that Google is both the chip designer *and* the AI company. So they get both sets of profits. Both Google and Nvidia contract TSMC for chips. Then Nvidia sells them at a huge profit. Then OpenAI (for example) buys them at that inflated rate and them puts them into production. So while Nvidia is "selling shovels", Google is making their own shovels and has their own mines.

> AI ... profits

Citation needed. But the vertical integration is likely valuable right now, especially with NVidia being supply constrained.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#176

Post by 7e »

m4r1k wrote: post #41

Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects. To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale…

No, not at all. If this were true Google would be killing it in MLPerf benchmarks, but they are not.

It’s better to have a faster, smaller network for model parallelism and a larger, slower one for data parallelism than a very large, but slower, network for everything. This is why NVIDIA wins.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#177

Post by 7e »

m4r1k wrote: post #89
villgax wrote: post #61

Earlier quoted context omitted.

100 times more chips for equivalent memory, sure.

Check the specs again. Per chip, TPU 7x has 192GB of HBM3e, whereas the NVIDIA B200 has 186GB. While the B200 wins on raw FP8 throughput (~9000 vs 4614 TFLOPs), that makes sense given NVIDIA has optimized for the single-chip game for over 20 years. But the bottleneck here isn't the chip—it's the domain size. NVIDIA's top-tier NVL72 tops out at an NVLink domain of 72 Blackwell GPUs. Meanwhile, Google is connecting 921…

Wow, no, not at all. It’s better to have a set of smaller, faster cliques connected by a slow network than a slower-than-clique flat network that connects everything. The cliques connected by a slow DCN can scale to arbitrary size. Even Google has had to resort to that for its biggest clusters.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#178

Post by calaphos »

m4r1k wrote: post #41

Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects. To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale…

That is comparing an all to all switched Nvlink fabric to a 3D torus for TPUs. Those are completely different network topologies with different tradeoffs.

For example the currently very popular Mixture of Experts architectures require a lot of all to all traffic (for expert parallelism) which works a lot better on the switched NVlink fabric as opposed where it doesn't need to traverse multiple links in the torus.

Re: TPUs vs. GPUs and why Google is positioned to win AI race in the long term

#179

Post by HarHarVeryFunny »

thesz wrote: post #173
HarHarVeryFunny wrote: post #77

Earlier quoted context omitted.

TPUs do include dedicated hardware, SparseCores, for sparse operations. https://docs.cloud.google.com/tpu/docs/system-architecture-t... https://openxla.org/xla/sparsecore

SparseCores appear to be block-sparse as opposed to element-sparse. They use 8- and 16-wide vectors to compute. Here's another inference-efficient architecture where TPUs are useless: https://arxiv.org/pdf/2210.08277 There is no matrix-vector multiplication. Parameters are estimated using Gumbel-Softmax. TPUs are of no use here. Inference is done bit-wise and most efficient inference is done after application of bool…

They are on their 7th generation now, so presumably the architecture is being updated as needs require.
Post reply on HN