Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ironwood: The first Google TPU for the age of inference

blog.google

Post reply on HN
Print view View original

61–70 of 186 posts

Re: Ironwood: The first Google TPU for the age of inference

#61

Post by christkv »

adrian_b wrote: post #46
fancyfredbot wrote: post #2

It looks amazing but I wish we could stop playing silly games with benchmarks. Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Why leave out TPUv6 in the comparison? Why compare fp64 flops in the El Capitan supercomputer to fp8 flops in the TPU pod when you know full well these are not comparable? [Edit: it turns out that El Capitan is actually faster when compared like f…

FP64 is more like 64 times harder than FP8. Actually the cost is even much higher, because the cost ratio is not much less than the square of the ratio between the sizes of the significands, which in this case is 52 bits / 4 bits = 13, and the square of 13 is 169.

Memory size and bandwidth goes up a lot right?

Re: Ironwood: The first Google TPU for the age of inference

#62

Post by bobim »

charcircuit wrote: post #16
fancyfredbot wrote: post #2

It looks amazing but I wish we could stop playing silly games with benchmarks. Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Why leave out TPUv6 in the comparison? Why compare fp64 flops in the El Capitan supercomputer to fp8 flops in the TPU pod when you know full well these are not comparable? [Edit: it turns out that El Capitan is actually faster when compared like f…

>Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Because end users want to use fp8. Why should architectural differences matter when the speed is what matters at the end of the day?

GP bikes are faster than dirt bikes, but not on dirt. The context has some influence here.

Re: Ironwood: The first Google TPU for the age of inference

#64

Post by fluidcruft »

llm_nerd wrote: post #37
fluidcruft wrote: post #29

This isn't anything anyone can purchase, is it? Who's the audience for this announcement?

The overwhelming majority of AI compute is by either the few bigs in their own products, or by third parties that rent out access to compute resources from those same bigs. Extremely few AI companies are buying their own GPU/TPU buildouts. Google says Ironwood will be available in the Google Cloud late this year, so it's relevant to just about anyone that rents AI compute, which is just about everyone in tech. Even i…

It just seems like John Deere putting out a press-release about about a new sparkplug that is only useful to John Deere and can maybe be used on rented John Deere harvesters when sharecropping on John Deere-owned fields using John Deere GMO crops. I just don't see what's appealing about any of it. Not only is it a walled garden, you can't even own anything and are completely dependent on the whims of John Deere to not bulldoze the entire field.

It just seems like if you build on Tensor then sure, you can go home, but Google will keep your ball.

Re: Ironwood: The first Google TPU for the age of inference

#65

Post by muxamilian »

_hark wrote: post #31

Can anyone comment on where efficiency gains come from these days at the arch level? I.e. not process-node improvements. Are there a few big things, many small things...? I'm curious what fruit are left hanging for fast SIMD matrix multiplication.

In-memory computing (analog or digital). Still doing SIMD matrix multiplication but using more efficient hardware: https://arxiv.org/html/2401.14428v1 https://www.nature.com/articles/s41565-020-0655-z

Re: Ironwood: The first Google TPU for the age of inference

#68

Post by kccqzy »

avrionov wrote: post #35
nehalem wrote: post #13

Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?

NVIDIA operates at 70% profit right now. Not paying that premium and having alternative to NVIDIA is beneficial. We just don't know how much.

I might be misremembering here, but Google's own AI models (Gemini) don't use NVIDIA hardware in any way, training or inference. Google bought a large number of NVIDIA hardware only for Google Cloud customers, not themselves.

Re: Ironwood: The first Google TPU for the age of inference

#69

Post by latchkey »

heymijo wrote: post #47
nehalem wrote: post #13

Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?

GPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for…

Cerebras (and Groq) has the problem of using too much die for compute and not enough for memory. Their method of scaling is to fan out the compute across more physical space. This takes more dc space, power and cooling, which is a huge issue. Funny enough, when I talked to Cerebras at SC24, they told me their largest customers are for training, not inference. They just market it as an inference product, which is even more confusing to me.

I wish I could say more about what AMD is doing in this space, but keep an eye on their MI4xx line.

Re: Ironwood: The first Google TPU for the age of inference

#70

Post by latchkey »

cheptsov wrote: post #38
latchkey wrote: post #19

Earlier quoted context omitted.

You said: "notice that it’s not compared to competitors" The article says: "When scaled to 9,216 chips per pod for a total of 42.5 Exaflops, Ironwood supports more than 24x the compute power of the world’s largest supercomputer – El Capitan – which offers just 1.7 Exaflops per pod." It is literally compared to a competitor.

I believe my original sentence was accurate. I was expecting the article to provide an objective comparison between TPUs and their main competitors. If you’re suggesting that El Capitan is the primary competitor, I’m not sure I agree, but I appreciate the perspective. Perhaps I was looking for other competitors, which is why I didn’t really pay attention to El Capitan.

Andrey, this is what I'm referring to: https://news.ycombinator.com/item?id=43632709
Post reply on HN