Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ironwood: The first Google TPU for the age of inference

blog.google

Post reply on HN
Print view View original

161–170 of 186 posts

Re: Ironwood: The first Google TPU for the age of inference

#161

Post by simondotau »

throwaway48476 wrote: post #150
simondotau wrote: post #146

Earlier quoted context omitted.

Seriously yes. I don’t want to own rapidly depreciating hardware if I don’t have to. I don’t want to own open source software and I’d prefer if culture was derived from and contributed to the public domain. I’m still fully on board with capitalism, but there are many instances where I’d prefer to replace physical acquisition with renting, or replacing corporate-made culture with public culture.

Nvidia cloud instances have competition. What happens when you get vendor lock in with TPU and have no exit plan? Arguably it's competition that drives value creation, not capitalism.

Just don’t let yourself get stuck behind vendor lock-in, or if you do, never let yourself feel trapped, because you never are.

Whenever assessing the work involved in building an integration, always assume you’ll be doing it twice. If that sounds like too much work then you shouldn’t have outsourced to begin with.

Re: Ironwood: The first Google TPU for the age of inference

#162

Post by marsten »

refulgentis wrote: post #104
miki123211 wrote: post #100

Earlier quoted context omitted.

Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?

https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://resea…

Google was changed by two things, neither of which were much fun. But very understandable.

One was the transition to a mature product line. In the early days it was about how do we do cool new things that will delight users: Gmail, Google Maps (Where 2), YouTube. The focus was on user growth and adoption.

Then growth saturated and the focus turned to profitability: Getting more value out of existing users and defending the business. That shift causes you to think very differently, and it's not as fun.

The second was changing market conditions. The web grew up, tech grew up, and the investment needed to make a competitive product skyrocketed. Google needed more wood behind fewer arrows and that meant reining in all the small teams running around doing kooky things. Again not fun, but understandable.

Re: Ironwood: The first Google TPU for the age of inference

#163

Post by literalAardvark »

dbmnt wrote: post #157
xnx wrote: post #110

Earlier quoted context omitted.

> what even is an AI data center A data center that runs significant AI training or inference loads. Non AI data centers are fairly commodity. Google's non-AI efficiency is not much better than Amazon or anyone else. Google is much more efficient at running AI workloads than anyone else.

> Google's non-AI efficiency is not much better than Amazon or anyone else. I don't think this is true. Google has long been a leader in efficiency. Look at the power usage effectiveness (PUE). A decade ago Google announced average PUEs around 1.12 while the industry average was closer to 2.0. From what I can tell they reported a 1.1 average fleet wide last year. They've been more transparent about this than any of t…

A decade ago seems like a very long time.

We've seen the rise of OSS Kubernetes and eBPF networking since, and a lot more that I don't have on-stack rn.

I wouldn't be surprised if everyone else had significantly closed the hardware utilization gap.

Re: Ironwood: The first Google TPU for the age of inference

#164

Post by nsteel »

dekhn wrote: post #128
nsteel wrote: post #102

Earlier quoted context omitted.

Wouldn't a 3d torus network have horrible performance with 9,216 nodes? And really horrible latency? I'd have assumed traditional spine-leaf would do better. But I must be wrong as they're claiming their latency is great here. Of course, they provide zero actual evidence of that. And I'll echo, what even is an AI data center, because we're still none the wiser.

A 3d torus is a tradeoff in terms of wiring complexity/cost and performance. When node counts get high you can't really have a pair of wires between all pairs of nodes, so if you don't use a torus you usually need a stack of switches/routers aggregating traffic. Those mid-level and top-level switch/routers get very expensive (high bandwidth cross-section) and the routing can get a bit painful. 3d torus has far fewer…

> most workloads would be nearest-neighbor

Thank you very much, that is the piece of the puzzle I was missing. Naively, it still seems (to me) far more hops for a 3d torus than a regular multi-level switch when you've got many thousands of nodes, but I can appreciate it could be much simpler routing. Although, I would guess in practice it requires something beyond the simplest routing solution to avoid congestion.

Re: Ironwood: The first Google TPU for the age of inference

#166

Post by nsteel »

mupuff1234 wrote: post #155
nickysielicki wrote: post #151

Earlier quoted context omitted.

Broadcom is fabless…

Oops.

You were right in this context. They have previously partnered with Broadcom to make their TPUs, rather than work directly with TSMC. But I read it's actually Mediatek instead for this generation.

Re: Ironwood: The first Google TPU for the age of inference

#170

Post by cubefox »

jmalicki wrote: post #77
vessenes wrote: post #42

Earlier quoted context omitted.

One big area the last two years has been algorithmic improvements feeding hardware improvements. Supercomputer folks use f64 for everything, or did. Most training was done at f32 four years ago. As algo teams have shown fp8 can be used for training and inference, hardware has updated to accommodate, yielding big gains. NB: Hobbyist, take all with a grain of salt

Unlike a lot of supercomputer algorithms, where fp error accumulates as you go, gradient descent based algorithms don't need as much precision since any fp errors will still show up at the next loss function calculation to be corrected, which allows you to make do with much lower precision.

Much lower indeed. Even Boolean functions (e.g. AND) are differentiable (though not exactly in the Newton/Leibniz sense) which can be used for backpropagation. They allow for an optimizer similar to stochastic gradient descent. There is a paper on it: https://arxiv.org/abs/2405.16339

It seems to me that floating point math (matrix multiplication) will over time mostly disappear from ML chips, as Boolean operations are much faster both in training an inference. But currently they are still optimized for FP rather than Boolean operations.

Post reply on HN