Its hard to be excited about hardware that will only exist in the cloud before shredding.
[flagged]
[dead]
Ironwood: The first Google TPU for the age of inference
51–60 of 186 posts
Re: Ironwood: The first Google TPU for the age of inference
#52Re: Ironwood: The first Google TPU for the age of inference
#53Earlier quoted context omitted.
> Who's the audience for this announcement? Probably whales who can afford to rent one from Google Cloud.
People with $3 are whales now? TPU prices are similar to other cloud resources.
Does anyone do anything useful with a $3 spend, or is it $3 X $manyManyHours?
Re: Ironwood: The first Google TPU for the age of inference
#54Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
GPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for…
The Groq interview was good too. Seems that the thought process is that companies like Groq/Cerebras can run the inference, and companies like Nvidia can keep/focus on their highly lucrative pretraining business.
Re: Ironwood: The first Google TPU for the age of inference
#55Re: Ironwood: The first Google TPU for the age of inference
#56Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
There are other ai/llm ‘specific’ chips out there, yes. But the thing about asics is that you need one for each *specific* task. Eventually we’ll hit an equilibrium but for now, the stuff that Cerebras is best at is not what TPUs are best at is not what GPUs are best at…
I don't even know if eventually we'll hit an equilibrium.
The end of Moore's law pretty much dictates specialization, it's just more apparent in fields without as much ossification first.
Re: Ironwood: The first Google TPU for the age of inference
#57Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
Anthropic is using Google TPUs. Also jointly working with Amazon on a data center using Amazon's custom AI chips. Also Google and Amazon are both investors in Anthropic.
https://www.datacenterknowledge.com/data-center-chips/ai-sta...
https://www.semafor.com/article/12/03/2024/amazon-announces-...
Re: Ironwood: The first Google TPU for the age of inference
#58This isn't anything anyone can purchase, is it? Who's the audience for this announcement?
> Who's the audience for this announcement? Probably whales who can afford to rent one from Google Cloud.
An on-demand v5e-1 is $1.2/h, it's pretty accessible.
The challenge is getting them to run efficiently, which typically involves learning JAX.
Re: Ironwood: The first Google TPU for the age of inference
#59Re: Ironwood: The first Google TPU for the age of inference
#60Its hard to be excited about hardware that will only exist in the cloud before shredding.
You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia?
This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.