Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ironwood: The first Google TPU for the age of inference

blog.google

Post reply on HN
Print view View original

81–90 of 186 posts

Re: Ironwood: The first Google TPU for the age of inference

#81

Post by woodson »

kleiba wrote: post #72
dgacmu wrote: post #18

Earlier quoted context omitted.

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…

> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!

So are CNNs, but I guess their popularity heavily increased at that time, to the point where it made sense to optimize the hardware for them.

Re: Ironwood: The first Google TPU for the age of inference

#82

Post by heymijo »

latchkey wrote: post #69
heymijo wrote: post #47

Earlier quoted context omitted.

GPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for…

Cerebras (and Groq) has the problem of using too much die for compute and not enough for memory. Their method of scaling is to fan out the compute across more physical space. This takes more dc space, power and cooling, which is a huge issue. Funny enough, when I talked to Cerebras at SC24, they told me their largest customers are for training, not inference. They just market it as an inference product, which is even…

> they told me their largest customers are for training, not inference

That is curious. Things are moving so quickly right now. I typed out a few speculative sentences then went ahead and asked an LLM.

Looks like Cerebras is responding to the market and pivoting towards a perceived strength of their product combined with the growth in inference, especially with the advent of reasoning models.

Re: Ironwood: The first Google TPU for the age of inference

#83

Post by hyhjtgh »

kleiba wrote: post #72
dgacmu wrote: post #18

Earlier quoted context omitted.

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…

> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!

RNN was of course well known at the at time, but they werent putting out state of the art numbers at that time.

Re: Ironwood: The first Google TPU for the age of inference

#84

Post by latchkey »

heymijo wrote: post #82
latchkey wrote: post #69

Earlier quoted context omitted.

Cerebras (and Groq) has the problem of using too much die for compute and not enough for memory. Their method of scaling is to fan out the compute across more physical space. This takes more dc space, power and cooling, which is a huge issue. Funny enough, when I talked to Cerebras at SC24, they told me their largest customers are for training, not inference. They just market it as an inference product, which is even…

> they told me their largest customers are for training, not inference That is curious. Things are moving so quickly right now. I typed out a few speculative sentences then went ahead and asked an LLM. Looks like Cerebras is responding to the market and pivoting towards a perceived strength of their product combined with the growth in inference, especially with the advent of reasoning models.

I wouldn't call it "pivoting" as much as "marketing".

Re: Ironwood: The first Google TPU for the age of inference

#86

Post by aseipp »

fluidcruft wrote: post #64
llm_nerd wrote: post #37

Earlier quoted context omitted.

The overwhelming majority of AI compute is by either the few bigs in their own products, or by third parties that rent out access to compute resources from those same bigs. Extremely few AI companies are buying their own GPU/TPU buildouts. Google says Ironwood will be available in the Google Cloud late this year, so it's relevant to just about anyone that rents AI compute, which is just about everyone in tech. Even i…

It just seems like John Deere putting out a press-release about about a new sparkplug that is only useful to John Deere and can maybe be used on rented John Deere harvesters when sharecropping on John Deere-owned fields using John Deere GMO crops. I just don't see what's appealing about any of it. Not only is it a walled garden, you can't even own anything and are completely dependent on the whims of John Deere to no…

The reality is that for large scale AI deployment there's only one criterion that matters: what is the total cost of ownership? If TPUs are 1/30th the total perf but 1/50th the total price, then they will be bought by customers. Basically that simple.

Most places using AI hardware don't actually want to expend massive amounts of capital to procure it and then shove it into racks somewhere and then manage it over its total lifetime. Hyperscalers like Google are also far, far ahead in things like DC energy efficiency, and at really large scale those energy costs are huge and have to be factored into the TCO. The long dominant cost of this stuff is all operational expenditures. Anyone running a physical AI cluster is going to have to consider this.

The walled garden stuff doesn't matter, because places demanding large-scale AI deployments (and actually willing to spend money on it) do not really have the same priorities as HN homelabbers who want to install inefficient 5090s so they can run Ollama.

Re: Ironwood: The first Google TPU for the age of inference

#87

Post by g42gregory »

And ... where could we get one? If they wouldn't sell it anyone, then is this a self-congratulation story? Why do we even need to know about this? If it propagates to the lower Gemini prices, fantastic. If not, then isn't it kind of irrelevant for the actual user experience?

Re: Ironwood: The first Google TPU for the age of inference

#88

Post by lordofgibbons »

g42gregory wrote: post #87

And ... where could we get one? If they wouldn't sell it anyone, then is this a self-congratulation story? Why do we even need to know about this? If it propagates to the lower Gemini prices, fantastic. If not, then isn't it kind of irrelevant for the actual user experience?

You can rent it on GCP in a few months

Re: Ironwood: The first Google TPU for the age of inference

#89

Post by jstummbillig »

g42gregory wrote: post #87

And ... where could we get one? If they wouldn't sell it anyone, then is this a self-congratulation story? Why do we even need to know about this? If it propagates to the lower Gemini prices, fantastic. If not, then isn't it kind of irrelevant for the actual user experience?

Well, with stocks and all, there is more that matters in the world than "actual user experience"

Re: Ironwood: The first Google TPU for the age of inference

#90

Post by g42gregory »

lordofgibbons wrote: post #88
g42gregory wrote: post #87

And ... where could we get one? If they wouldn't sell it anyone, then is this a self-congratulation story? Why do we even need to know about this? If it propagates to the lower Gemini prices, fantastic. If not, then isn't it kind of irrelevant for the actual user experience?

You can rent it on GCP in a few months

Good point. At what prices per GB/TOPS? Better be lower than the existing TPUs ... That's what I care about.
Post reply on HN