Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ironwood: The first Google TPU for the age of inference

blog.google

Post reply on HN
Print view View original

121–130 of 186 posts

Re: Ironwood: The first Google TPU for the age of inference

#121

Post by xadhominemx »

nsteel wrote: post #102
xnx wrote: post #97

Earlier quoted context omitted.

> Because of the TPUs, or due to other factors? Google does many pieces of the data center better. Google TPUs use 3D torus networking and are liquid cooled. > What even is an AI data center? Being newer, AI installations have more variations/innovation than traditional data centers. Google's competitors have not yet adopted all of Google's advances. > are the GPU/TPU boxes in a different building than the others? No…

Wouldn't a 3d torus network have horrible performance with 9,216 nodes? And really horrible latency? I'd have assumed traditional spine-leaf would do better. But I must be wrong as they're claiming their latency is great here. Of course, they provide zero actual evidence of that. And I'll echo, what even is an AI data center, because we're still none the wiser.

It’s data center with much higher power density. We’re talking about 100 going to 1,000 kw/rack vs 20 kw/rack for a traditional data center. Requiring much different cooling a power delivery.

Re: Ironwood: The first Google TPU for the age of inference

#123

Post by summerlight »

claytonjy wrote: post #95
xnx wrote: post #50

Earlier quoted context omitted.

Google has a significant advantage over other hyperscalers because Google's AI data centers are much more compute cost efficient (capex and opex).

Because of the TPUs, or due to other factors? What even is an AI data center? are the GPU/TPU boxes in a different building than the others?

Lots of other factors. I suspect this is one of the reasons why Google cannot offer TPU hardware itself out of their cloud service. A significant chunk of TPU efficiency can be attributed external factors which customers cannot easily replicate.

Re: Ironwood: The first Google TPU for the age of inference

#124

Post by joshuamorton »

throwaway48476 wrote: post #98
crazygringo wrote: post #60

Earlier quoted context omitted.

You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia? This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.

[flagged]

Google's been doing custom ML accelerators for 10 years now, and (depending on how much you're willing to stretch the definition) has been doing them in consumer hardware for soon to be five years (the Google Tensor chips in pixel phones).

Re: Ironwood: The first Google TPU for the age of inference

#125

Post by asdfman123 »

mycall wrote: post #59
p_j_w wrote: post #30

Earlier quoted context omitted.

I think this article is for Wall Street, not Silicon Valley.

Bad timing as I think Wall Street is preoccupied at the moment.

Oh, believe me, they are very much paying attention to tech stocks right now.

Re: Ironwood: The first Google TPU for the age of inference

#126

Post by fluidcruft »

aseipp wrote: post #86
fluidcruft wrote: post #64

Earlier quoted context omitted.

It just seems like John Deere putting out a press-release about about a new sparkplug that is only useful to John Deere and can maybe be used on rented John Deere harvesters when sharecropping on John Deere-owned fields using John Deere GMO crops. I just don't see what's appealing about any of it. Not only is it a walled garden, you can't even own anything and are completely dependent on the whims of John Deere to no…

The reality is that for large scale AI deployment there's only one criterion that matters: what is the total cost of ownership? If TPUs are 1/30th the total perf but 1/50th the total price, then they will be bought by customers. Basically that simple. Most places using AI hardware don't actually want to expend massive amounts of capital to procure it and then shove it into racks somewhere and then manage it over its…

At large scales why shouldn't it matter whether you're beholden to Google's cloud only vs having options to use AWS or Oracle or Azure etc. There's maybe an argument to be made about price and efficiency of Google's data centers, but Google's cloud is far from notably cheaper than alternatives (to put it mildly) so that's a moot point if there's any efficiencies to be had Google's pocketing it themselves. I just don't see why anyone should care about this chip except Google themselves. It would be a different story if we were talking about a chip that had the option of being available in non-Google data centers.

Re: Ironwood: The first Google TPU for the age of inference

#127

Post by dekhn »

miki123211 wrote: post #100
dgacmu wrote: post #18

Earlier quoted context omitted.

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…

Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?

As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed.

The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were a good alternative. This was around the time ImageNet blew up, and some eng were stuffing multiple GPUs in their dev boxes to demonstrate training speeds on tasks like voice recognition.

Sibyl was a heavy user of embeddings before there was any real custom ASIC support for that and there was an add-on for TPUs called barnacore to give limited embedding support (embeddings are very useful for maximizing profit through ranking).

Re: Ironwood: The first Google TPU for the age of inference

#128

Post by dekhn »

nsteel wrote: post #102
xnx wrote: post #97

Earlier quoted context omitted.

> Because of the TPUs, or due to other factors? Google does many pieces of the data center better. Google TPUs use 3D torus networking and are liquid cooled. > What even is an AI data center? Being newer, AI installations have more variations/innovation than traditional data centers. Google's competitors have not yet adopted all of Google's advances. > are the GPU/TPU boxes in a different building than the others? No…

Wouldn't a 3d torus network have horrible performance with 9,216 nodes? And really horrible latency? I'd have assumed traditional spine-leaf would do better. But I must be wrong as they're claiming their latency is great here. Of course, they provide zero actual evidence of that. And I'll echo, what even is an AI data center, because we're still none the wiser.

A 3d torus is a tradeoff in terms of wiring complexity/cost and performance. When node counts get high you can't really have a pair of wires between all pairs of nodes, so if you don't use a torus you usually need a stack of switches/routers aggregating traffic. Those mid-level and top-level switch/routers get very expensive (high bandwidth cross-section) and the routing can get a bit painful. 3d torus has far fewer cables, and the routing can be really simple ("hop vertically until you reach your row, then hop horizontally to read your node"), and the wrap-around connections are nice.

That said, the torus approach was a gamble that most workloads would be nearest-neighbor, and allreduce needs extra work to optimize.

An AI data center tends to have enormous power consumption and cooling capabilities, with less disk, and slightly different networking setups. But really it just means "this part of the warehouse has more ML chips than disks"

Re: Ironwood: The first Google TPU for the age of inference

#130

Post by wyager »

epolanski wrote: post #48
no_wizard wrote: post #10

Some honest competition in the chip space in the machine learning race! Genuinely interested to see how this ends up playing out. Nvidia seemed 'untouchable' for so long in this space that its nice to see things get shaken up. I know they aren't selling the TPU as boxed units, but still, even as hardware that backs GCP services and what not, its interesting to see how it'll shake out!

> Nvidia seemed 'untouchable' for so long in this space that its nice to see things get shaken up. Did it? Both Mistral's LeChat (running on Cerebras) and Google's Gemini (running on Tensors) have clearly showed ages ago Nvidia had no advantage at all in inference. The hundreds of billions spent in hardware till now focused on training, but inference is in the long run gonna get the lion share of the work.

> but inference is in the long run gonna get the lion share of the work.

I'm not sure - might not the equilibrium state be that we are constantly fine-tuning models with the latest data (e.g. social media firehose)?

Post reply on HN