101–110 of 186 posts
Earlier quoted context omitted.
Because of the TPUs, or due to other factors? What even is an AI data center? are the GPU/TPU boxes in a different building than the others?
> Because of the TPUs, or due to other factors? Google does many pieces of the data center better. Google TPUs use 3D torus networking and are liquid cooled. > What even is an AI data center? Being newer, AI installations have more variations/innovation than traditional data centers. Google's competitors have not yet adopted all of Google's advances. > are the GPU/TPU boxes in a different building than the others? No…
And I'll echo, what even is an AI data center, because we're still none the wiser.
The first specifically designed for inference? Wasn’t the original TPU inference only?
Earlier quoted context omitted.
Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…
Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?
I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it.
The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-stru...
It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it.
FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)
Earlier quoted context omitted.
Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply un…
Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?
[1]: https://cloud.google.com/blog/products/ai-machine-learning/g...
[2]: https://github.com/rikhuijzer/improv/blob/master/runs/2018-1...
Earlier quoted context omitted.
In-memory computing (analog or digital). Still doing SIMD matrix multiplication but using more efficient hardware: https://arxiv.org/html/2401.14428v1 https://www.nature.com/articles/s41565-020-0655-z
This is very interesting, but not what the Ironside TPU is doing. The blog post says that the TPU uses conventional HBM RAM.
https://semiengineering.com/speeding-down-memory-lane-with-c...
Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
GPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for…
Earlier quoted context omitted.
You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia? This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.
[flagged]
Like graviton at AWS its as much of a negotiation tool as it is a technical solution, letting them push harder with NVIDIA on pricing because they have a backup option.
Earlier quoted context omitted.
> Because of the TPUs, or due to other factors? Google does many pieces of the data center better. Google TPUs use 3D torus networking and are liquid cooled. > What even is an AI data center? Being newer, AI installations have more variations/innovation than traditional data centers. Google's competitors have not yet adopted all of Google's advances. > are the GPU/TPU boxes in a different building than the others? No…
Wouldn't a 3d torus network have horrible performance with 9,216 nodes? And really horrible latency? I'd have assumed traditional spine-leaf would do better. But I must be wrong as they're claiming their latency is great here. Of course, they provide zero actual evidence of that. And I'll echo, what even is an AI data center, because we're still none the wiser.
A data center that runs significant AI training or inference loads. Non AI data centers are fairly commodity. Google's non-AI efficiency is not much better than Amazon or anyone else. Google is much more efficient at running AI workloads than anyone else.