I wonder if these chips might contribute towards advancements for the Coral TPU chips?
Ironwood: The first Google TPU for the age of inference
131–140 of 186 posts
Re: Ironwood: The first Google TPU for the age of inference
#132Earlier quoted context omitted.
GPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for…
Cerebras (and Groq) has the problem of using too much die for compute and not enough for memory. Their method of scaling is to fan out the compute across more physical space. This takes more dc space, power and cooling, which is a huge issue. Funny enough, when I talked to Cerebras at SC24, they told me their largest customers are for training, not inference. They just market it as an inference product, which is even…
Thank you for sharing this perspective — really insightful. I’ve been reading up on Groq’s architecture and was under the impression that their chips dedicate a significant portion of die area to on-chip SRAM (around 220MiB per chip, if I recall correctly), which struck me as quite generous compared to typical accelerators.
From die shots and materials I’ve seen, it even looks like ~40% of the die might be allocated to memory [1]. Given that, I’m curious about your point on “not enough die for memory” — is it a matter of absolute capacity still being insufficient for current model sizes, or more about the area-bandwidth tradeoff being unbalanced for inference workloads? Or perhaps something else entirely?
I’d love to understand this design tension more deeply, especially from someone with a high-level view of real-world deployments. Thanks again.
[1] Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads — Fig. 5. Die photo of 14nm ASIC implementation of the Groq TSP. https://groq.com/wp-content/uploads/2024/02/2020-Isca.pdf
Re: Ironwood: The first Google TPU for the age of inference
#133Re: Ironwood: The first Google TPU for the age of inference
#134How is API story for these devices? Are the drivers mainlined in Linux? Is there a specific API you use to code for them? How does the instance you rent on Google Cloud look and what does that have for software?
They have out of tree drivers. If they don't ship the hardware to end users, it's not clear upstream (Linux kernel) would want them.
Re: Ironwood: The first Google TPU for the age of inference
#135It looks amazing but I wish we could stop playing silly games with benchmarks. Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Why leave out TPUv6 in the comparison? Why compare fp64 flops in the El Capitan supercomputer to fp8 flops in the TPU pod when you know full well these are not comparable? [Edit: it turns out that El Capitan is actually faster when compared like f…
[deleted]
Re: Ironwood: The first Google TPU for the age of inference
#136Who manufactures their TPUs? Wondering whether they're US-made, and therefore immune to the tariff craziness.
Re: Ironwood: The first Google TPU for the age of inference
#137Re: Ironwood: The first Google TPU for the age of inference
#138Earlier quoted context omitted.
Wouldn't a 3d torus network have horrible performance with 9,216 nodes? And really horrible latency? I'd have assumed traditional spine-leaf would do better. But I must be wrong as they're claiming their latency is great here. Of course, they provide zero actual evidence of that. And I'll echo, what even is an AI data center, because we're still none the wiser.
A 3d torus is a tradeoff in terms of wiring complexity/cost and performance. When node counts get high you can't really have a pair of wires between all pairs of nodes, so if you don't use a torus you usually need a stack of switches/routers aggregating traffic. Those mid-level and top-level switch/routers get very expensive (high bandwidth cross-section) and the routing can get a bit painful. 3d torus has far fewer…
Was that gamble wrong? I thought all LLM training workloads do collectives that involve all nodes (all-gather, reduce-scatter).
Re: Ironwood: The first Google TPU for the age of inference
#139Earlier quoted context omitted.
You can't get excited about lower prices for your cloud GPU workloads thanks to the competition it brings to Nvidia? This benefits everyone, even if you don't use Google Cloud, because of the competition it introduces.
I like owning things
Oh, you own the generator for you GPU as well?
Re: Ironwood: The first Google TPU for the age of inference
#140Earlier quoted context omitted.
Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?
As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed. The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were…
What I've heard is that the extrapolation of compute needed so many additional CPU servers to keep running the existing workload types that it obviously justified dedicated hardware. Same for video encoding accelerators[1].
[1]: https://research.google/pubs/warehouse-scale-video-accelerat...