> 35x increase in AI inference performance compared to AMD Instinct MI300 Series
Even for marketing claims that’s pretty wild.
Still lots of trajectory left in just scale up plan it seems
AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
31–40 of 53 posts
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#32The claim that the next generation would be 35x faster, felt like an "Osborne moment" to me, but if demand is robust enough...
(Re: Osborne effect) It's going to be released in 2 years. Rarely can businesses wait that long, they're going to be ordering the MI300 now.
Or they're trying to distract attention from the fact that they've already sold out 100% of the fab capacity available to produce these chips for the next two years.
So really, they lose nothing. They've already booked sales of everything there is to sell. So might as well now turn attention to those who might be customers two years from now, and make them feel like the wait will be worth it.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#33Earlier quoted context omitted.
Shilling is ok on a topic directly related to my business. You're right on the Osborne effect though! Thanks for that. We are definitely not doing that. To clarify: When we started, MI300x was not officially announced yet, so we were planning on buying MI250's. Due to everything taking longer than expected around starting the business and receiving funding, by the time we had money in the bank, it was time to buy MI3…
Since we're on the topic of your business: I am training a decent amount of neural nets nowadays (mostly, around the new-gen robotics policies) and use vast.ai instances with 8x RTX 4090 cards. I've been interested to give 8x MI300x a try, as they are supposed to be cheaper per FLOPs, but it looks like your service does not provide on-demand pay-per-second instances. Any plans to change that?
I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand.
It kind of makes sense since their history is only supporting the high end GPUs in their HPC solutions, where they don't use VM's. They've committed to us directly that they will fix this issue.
I updated our pricing page to note this.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#34Such a weird & cruel modernity, where these releases are purely in the abstract. No, you still won't be able to buy a MI300X in Q4 2024. The enhanced edition will absolutely not be available. (I miss the old PC era where the world at large was benefiting in tandem from new things happening (or falling behind from not adapting)).
The focus is on hyperscalers and cloud service providers now. Even Groq stopped selling to retail.
Welcome (back) to the age of the mainframe.
Except we call it "cloudframe" now.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#35Such a weird & cruel modernity, where these releases are purely in the abstract. No, you still won't be able to buy a MI300X in Q4 2024. The enhanced edition will absolutely not be available. (I miss the old PC era where the world at large was benefiting in tandem from new things happening (or falling behind from not adapting)).
> No, you still won't be able to buy a MI300X in Q4 2024. they're 15k - who exactly is disappointed they won't be able to buy one?
> who exactly is disappointed they won't be able to buy one?
HPC centers and research clusters.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#36The claim that the next generation would be 35x faster, felt like an "Osborne moment" to me, but if demand is robust enough...
In AI, that doesn't sound too surprising to me right now.
I just experiment with some local LLMs, but the differences are pretty huge:
Llama 3 8B, Raspberry Pi 5: 2-3 Tokens/second (but it works!)
Llama 3 8B, RTX 4080: ~60 Tokens/second
Llama 3 8B, groq.com LPU, ~1300 Tokens/second
Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second
Llama 3 70B, groq.com LPU, ~330 Tokens/second
There seem to be huge gaps between CPU, GPU and specialized inference ASICs. I'm guessing that right now there aren't many genius-level architecture breakthroughs, and that it's more about how much memory and silicon real estate you're willing to dedicate to AI inference.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#37> 35x increase in AI inference performance compared to AMD Instinct MI300 Series Even for marketing claims that’s pretty wild. Still lots of trajectory left in just scale up plan it seems
I think there is a close limit considering most of these gains are coming from the reduced memory bandwidth consumption that comes with the smaller data types. This would line up with Nvidia’s crazy graph from yesterday where data types were specified.
How much lower can these go though? 2bit? 1.58bit? 1bit? It seems that these massive gains have a very hard stop to gains that AMD and Nvidia will use to raise their stock price before it all comes to a sudden end.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#38AMD comparison:
8x AMD MI300X (192GB, 750W) GPU
8x H100 (80GB, 700W) GPU
8x H100 NVL (188GB, Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#39The claim that the next generation would be 35x faster, felt like an "Osborne moment" to me, but if demand is robust enough...
In AI, that doesn't sound too surprising to me right now. I just experiment with some local LLMs, but the differences are pretty huge: Llama 3 8B, Raspberry Pi 5: 2-3 Tokens/second (but it works!) Llama 3 8B, RTX 4080: ~60 Tokens/second Llama 3 8B, groq.com LPU, ~1300 Tokens/second Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second Llama 3 70B, groq.com LPU, ~330 Tokens/second There seem to be huge gaps between CPU, GPU and…
What quantization levels did you use?
I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#40Earlier quoted context omitted.
In AI, that doesn't sound too surprising to me right now. I just experiment with some local LLMs, but the differences are pretty huge: Llama 3 8B, Raspberry Pi 5: 2-3 Tokens/second (but it works!) Llama 3 8B, RTX 4080: ~60 Tokens/second Llama 3 8B, groq.com LPU, ~1300 Tokens/second Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second Llama 3 70B, groq.com LPU, ~330 Tokens/second There seem to be huge gaps between CPU, GPU and…
What quantization levels did you use? I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart.
> I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart.
To my knowledge this isn't (absolutely) publicly known but users on /r/LocalLLaMA and elsewhere have provided some pretty clear examples that Groq is almost certainly quantized. Which makes sense considering their memory situation...
An entire GroqRack (42U cabinet) has 14GB of RAM which means it likely can't even reasonably run llama3 8b in BF16/FP16. Let alone 70b, Mixtral, etc.
The amount of hardware required to run their public-facing hosted product likely takes up an obscene amount of floor space, even in int4. Their docs for GrowFlow describe int8 quantization but their toolkit is heavily dependent on ONNX, which has had recent tremendous work in terms of different post training quantization strategies and precisions.
However, the power efficiency vs performance is very good, potentially to the point of being able to use very cheap datacenter/co-location space that isn't capable of meeting the power and (air) cooling densities of datacenter AMD and Nvidia GPU products.
Interestingly I have access to a GroqRack system that I'm hoping to be able to spend some time on this week.