Cool. But does it support CUDA?
Ironwood: The first Google TPU for the age of inference
91–100 of 186 posts
Re: Ironwood: The first Google TPU for the age of inference
#92How is API story for these devices? Are the drivers mainlined in Linux? Is there a specific API you use to code for them? How does the instance you rent on Google Cloud look and what does that have for software?
XLA (Accelerated Linear Algebra) [1] is likely the library that you'll want to use to code for these machines.
TensorFlow, PyTorch, and Jax all support XLA on the backend.
[1]: https://openxla.org/
Re: Ironwood: The first Google TPU for the age of inference
#93Its hard to be excited about hardware that will only exist in the cloud before shredding.
You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia? This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.
I like owning things
Re: Ironwood: The first Google TPU for the age of inference
#94Can anyone comment on where efficiency gains come from these days at the arch level? I.e. not process-node improvements. Are there a few big things, many small things...? I'm curious what fruit are left hanging for fast SIMD matrix multiplication.
In-memory computing (analog or digital). Still doing SIMD matrix multiplication but using more efficient hardware: https://arxiv.org/html/2401.14428v1 https://www.nature.com/articles/s41565-020-0655-z
This is very interesting, but not what the Ironside TPU is doing. The blog post says that the TPU uses conventional HBM RAM.
Re: Ironwood: The first Google TPU for the age of inference
#95Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
Google has a significant advantage over other hyperscalers because Google's AI data centers are much more compute cost efficient (capex and opex).
Because of the TPUs, or due to other factors?
What even is an AI data center? are the GPU/TPU boxes in a different building than the others?
Re: Ironwood: The first Google TPU for the age of inference
#96Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
Nvidia has ~60% margins in their datacenter chips. So TPU's have quite a bit of headroom to save google money without being as good as Nvidia GPU's. No one else has access to anything similar, Amazon is just starting to scale their Trainium chip.
Microsoft has the MAIA 100 as well. No comment on their scale/plans though.
Re: Ironwood: The first Google TPU for the age of inference
#97Earlier quoted context omitted.
Google has a significant advantage over other hyperscalers because Google's AI data centers are much more compute cost efficient (capex and opex).
Because of the TPUs, or due to other factors? What even is an AI data center? are the GPU/TPU boxes in a different building than the others?
> Because of the TPUs, or due to other factors?
Google does many pieces of the data center better. Google TPUs use 3D torus networking and are liquid cooled.
> What even is an AI data center?
Being newer, AI installations have more variations/innovation than traditional data centers. Google's competitors have not yet adopted all of Google's advances.
> are the GPU/TPU boxes in a different building than the others?
Not that I've read. They are definitely bringing on new data centers, but I don't know if they are initially designed for pure-AI workloads.
Re: Ironwood: The first Google TPU for the age of inference
#98Its hard to be excited about hardware that will only exist in the cloud before shredding.
You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia? This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.
[flagged]
Re: Ironwood: The first Google TPU for the age of inference
#99Can anyone buy them?
Re: Ironwood: The first Google TPU for the age of inference
#100The first specifically designed for inference? Wasn’t the original TPU inference only?
Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…
Wow, you guys needed a custom ASIC for inference before CNNs were even invented?
What were the use cases like back then?