41–50 of 186 posts
Can anyone comment on where efficiency gains come from these days at the arch level? I.e. not process-node improvements. Are there a few big things, many small things...? I'm curious what fruit are left hanging for fast SIMD matrix multiplication.
Can anyone comment on where efficiency gains come from these days at the arch level? I.e. not process-node improvements. Are there a few big things, many small things...? I'm curious what fruit are left hanging for fast SIMD matrix multiplication.
NB: Hobbyist, take all with a grain of salt
These continue to be mostly for bragging rights and strategic safety I think. I bet they are not on premium processor nodes; If I worked at GOOG I’d probably think about these as competitive insurance vis-a-vis NVIDIA — total costs of chip team, software, tape outs, and increased data center energy use probably wipe out any savings from not buying NV, but you are 100% not beholden to Jensen.
Earlier quoted context omitted.
Also, there is no such thing as a "El Capitan pod". The quoted number is for the entire supercomputer. My impression from this is that they are too scared to say that their TPU pod is equivalent to 60 GB200 NVL72 racks in terms of fp8 flops. I can only assume that they need way more than 60 racks and they want to hide this fact.
A max-spec v5p deployment, at least the biggest one they'll let you rent, occupies 140 racks, for reference.
So $68k per hour or $27 million per month.
Get 55% off with 3 year commitment.
This isn't anything anyone can purchase, is it? Who's the audience for this announcement?
It looks amazing but I wish we could stop playing silly games with benchmarks. Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Why leave out TPUv6 in the comparison? Why compare fp64 flops in the El Capitan supercomputer to fp8 flops in the TPU pod when you know full well these are not comparable? [Edit: it turns out that El Capitan is actually faster when compared like f…
Actually the cost is even much higher, because the cost ratio is not much less than the square of the ratio between the sizes of the significands, which in this case is 52 bits / 4 bits = 13, and the square of 13 is 169.
Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
Why?
For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word.
GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for every single word it creates. This architectural choice, is an advantage for graphics processing where large amounts of data needs to be stored but not necessarily accessed as rapidly for every single computation. It's a liability in inference where quick and frequent data access is critical.
Listening to Andrew Feldman of Cerebras [0] is what helped me grok the differences. Caveat, he is a founder/CEO of a company that sells hardware for AI inference, so the guy is talking his book.
[0] https://www.youtube.com/watch?v=MW9vwF7TUI8&list=PLnJFlI3aIN...
Some honest competition in the chip space in the machine learning race! Genuinely interested to see how this ends up playing out. Nvidia seemed 'untouchable' for so long in this space that its nice to see things get shaken up. I know they aren't selling the TPU as boxed units, but still, even as hardware that backs GCP services and what not, its interesting to see how it'll shake out!
Did it?
Both Mistral's LeChat (running on Cerebras) and Google's Gemini (running on Tensors) have clearly showed ages ago Nvidia had no advantage at all in inference.
The hundreds of billions spent in hardware till now focused on training, but inference is in the long run gonna get the lion share of the work.
It would be awesome for things like homelabs (to run Frigate NVR, Immich ML tasks or the Home Assistant LLM).
Not knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?