171–180 of 186 posts
Earlier quoted context omitted.
> but inference is in the long run gonna get the lion share of the work. I'm not sure - might not the equilibrium state be that we are constantly fine-tuning models with the latest data (e.g. social media firehose)?
Head of groq said that in his experience at google training was less than 10% of compute.
Earlier quoted context omitted.
I love to hate on google, but I suspect this is strategic enough that they wont kill it. Like graviton at AWS its as much of a negotiation tool as it is a technical solution, letting them push harder with NVIDIA on pricing because they have a backup option.
Google has done stuff primarily for negotiation purposes (e.g. POWER9 chips) but TPU ain't one. It's not a backup option or presumed "inferior solution" to NVIDIA. Their entire ecosystem is TPU-first.
Why doesnt google offer the most advanced voice technology when they offer a playback version, it still sounds like the most basic text to voice.
Earlier quoted context omitted.
A 3d torus is a tradeoff in terms of wiring complexity/cost and performance. When node counts get high you can't really have a pair of wires between all pairs of nodes, so if you don't use a torus you usually need a stack of switches/routers aggregating traffic. Those mid-level and top-level switch/routers get very expensive (high bandwidth cross-section) and the routing can get a bit painful. 3d torus has far fewer…
Was that gamble wrong? I thought all LLM training workloads do collectives that involve all nodes (all-gather, reduce-scatter).
Earlier quoted context omitted.
As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed. The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were…
What I've heard is that the extrapolation of compute needed so many additional CPU servers to keep running the existing workload types that it obviously justified dedicated hardware. Same for video encoding accelerators[1]. [1]: https://research.google/pubs/warehouse-scale-video-accelerat...
Another part that was left out was that Google did not make truly high speed (low-latency) networking and so many of their CPU jobs had to be engineered around slow networks to maintain high utilization and training speed. Google basically ended up internally relearning the lessons that HPC and supercomputing communities had already established over decades.
Earlier quoted context omitted.
Why not? If we line up to race. You can't say why compare v8 to v6 turbo or electric engine. It's a race, the drive train doesn't matter. Who gets to the finish line first? No one is shopping for GPU by fp8, fp16, fp32, fp64. It's all about cost/performance factor. 8 bits is as good as 32bits, great performance is even been pulled out of 4 bits...
This is like saying I'm faster because I ran (a mile) in 8 minutes whereas it took you 15 minutes (to run two miles).
Dropping the analogy: f64 multiplication is a lot harder than f8 multiplication, but for ML tasks it's just not needed. f8 multiplication hardware is the right tool for the job.
Earlier quoted context omitted.
Wow, you guys needed a custom ASIC for inference before CNNs were even invented ? What were the use cases like back then?
As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed. The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were…
Ah, the days when you, as a tech company employee, could call a service "SmartASS" and get away with it...
Earlier quoted context omitted.
Google has done stuff primarily for negotiation purposes (e.g. POWER9 chips) but TPU ain't one. It's not a backup option or presumed "inferior solution" to NVIDIA. Their entire ecosystem is TPU-first.
Was Gemini 2.5 trained on TPUs thought? I seem to be struggling to find that information. Wouldn't they want to mention it in every press release?
P.S. I found an on-the-record statement re Gemini 1.0 on TPU:
"We trained Gemini 1.0 at scale on our AI-optimized infrastructure using Google’s in-house designed Tensor Processing Units (TPUs) v4 and v5e. And we designed it to be our most reliable and scalable model to train, and our most efficient to serve."
Earlier quoted context omitted.
Because it is a public company that aims to maximise shareholder value and thus the value of it's stock. Since value is largely evaluated by perception, if you can convince people your product is better than it is, your stock valuation, at least in the short term will be higher. Hence Tesla saying FSD and robo-taxis are 1 year away, the fusion companies saying fusion is closer than it is etc.... Nvidia, AMD, apple an…
I understand the value of perception. A big part of my issue here is that they've really messed up the misleading benchmarks. They've failed to compare to the most obvious alternative, which is Nvidia GPUs. They look like they've got something to hide, not like they're ahead. They've needlessly made their own current products look bad in comparison to this one understating the long-standing advantage TPUs have given…
Earlier quoted context omitted.
Thank you for sharing this perspective — really insightful. I’ve been reading up on Groq’s architecture and was under the impression that their chips dedicate a significant portion of die area to on-chip SRAM (around 220MiB per chip, if I recall correctly), which struck me as quite generous compared to typical accelerators. From die shots and materials I’ve seen, it even looks like ~40% of the die might be allocated…
> is it a matter of absolute capacity still being insufficient for current model sizes This. Additionally, models aren't getting smaller, they are getting bigger and to be useful to a wider range of users, they also need more context to go off of, which is even more memory. Previously: https://news.ycombinator.com/item?id=42003823 It could be partially the DC, but look at the rack density... to get to an equal amount…
What I’m still trying to understand is the economics.
From this benchmark: https://artificialanalysis.ai/models/llama-4-scout/providers...
Groq seems to offer near lowest prices per million tokens and the near fastest end to end response times. That’s surprising because in my understanding, speed(latency) and the cost are trade-offs.
So I’m wondering: Why can’t GPU-based providers can't offer cheaper but slower(high-latency) APIs? Or do you think Groq/Cerebras are pricing much below cost (loss-leader style)?