Ironwood: The first Google TPU for the age of inference

Post by latchkey » Sat, Apr 12, 2025, 3:19 AM UTC

usatie wrote: post #180

latchkey wrote: post #143

Earlier quoted context omitted.

> is it a matter of absolute capacity still being insufficient for current model sizes This. Additionally, models aren't getting smaller, they are getting bigger and to be useful to a wider range of users, they also need more context to go off of, which is even more memory. Previously: https://news.ycombinator.com/item?id=42003823 It could be partially the DC, but look at the rack density... to get to an equal amount…

Thanks for the links — I went through all of them (took me a while). The point about rack density differences between SRAM-based systems like Cerebras or Groq and GPU clusters is now clear to me. What I’m still trying to understand is the economics. From this benchmark: https://artificialanalysis.ai/models/llama-4-scout/providers... Groq seems to offer near lowest prices per million tokens and the near fastest end to…

Post by yencabulator » Wed, Apr 16, 2025, 3:31 PM UTC

theptip wrote: post #103

nharada wrote: post #12

The first specifically designed for inference? Wasn’t the original TPU inference only?

The phrasing is very precise here, it’s the first TPU for _the age of inference_, which is a novel marketing term they have defined to refer to CoT and Deep Research.

Post by simondotau » Thu, Apr 17, 2025, 12:33 PM UTC

throwaway48476 wrote: post #182

simondotau wrote: post #161

Earlier quoted context omitted.

Just don’t let yourself get stuck behind vendor lock-in, or if you do, never let yourself feel trapped, because you never are. Whenever assessing the work involved in building an integration, always assume you’ll be doing it twice. If that sounds like too much work then you shouldn’t have outsourced to begin with.

Switching costs are a thing everywhere in the economy. Measure twice cut once. Planning on doing the work twice is asinine.