Is the software stack working (for practical use)?
AMD still has to prove themselves in this.
AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
41–50 of 53 posts
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#42[deleted]
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#43Earlier quoted context omitted.
What quantization levels did you use? I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart.
> I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart. To my knowledge this isn't (absolutely) publicly known but users on /r/LocalLLaMA and elsewhere have provided some pretty clear examples that Groq is almost certainly quantized. Which makes sense considering their memory situation... An entire GroqRack (42U cabinet) has 14GB of RAM which means it likely can…
Ah TIL, thanks for the insights!
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#44The claim that the next generation would be 35x faster, felt like an "Osborne moment" to me, but if demand is robust enough...
In AI, that doesn't sound too surprising to me right now. I just experiment with some local LLMs, but the differences are pretty huge: Llama 3 8B, Raspberry Pi 5: 2-3 Tokens/second (but it works!) Llama 3 8B, RTX 4080: ~60 Tokens/second Llama 3 8B, groq.com LPU, ~1300 Tokens/second Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second Llama 3 70B, groq.com LPU, ~330 Tokens/second There seem to be huge gaps between CPU, GPU and…
> Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second
How much RAM is required for this result? It's quite impressive that it even works as well as it does.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#45Earlier quoted context omitted.
> Crypto miners no longer buy up large quantities of GPUs. That's news to crypto miners.
I no longer have the 150,000 GPUs I was mining with. The only ones left standing are the small ones and most of them are just using what they already purchased. There is no liquidity in the shitcoins to be able to support daily large dumps of their tokens and unless you have almost free power, there is zero profitability.
You need to start a blog latchkey, I see you on here in all sorts of different contexts and it’s always interesting to see what you’re up to these days!
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#46Earlier quoted context omitted.
Since we're on the topic of your business: I am training a decent amount of neural nets nowadays (mostly, around the new-gen robotics policies) and use vast.ai instances with 8x RTX 4090 cards. I've been interested to give 8x MI300x a try, as they are supposed to be cheaper per FLOPs, but it looks like your service does not provide on-demand pay-per-second instances. Any plans to change that?
I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. It kind of makes sense since their…
>I updated our pricing page to note this
Nice job on your website!
Sincerely,
Someone who recently gave you a hard time for not having info on your website.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#47Earlier quoted context omitted.
Since we're on the topic of your business: I am training a decent amount of neural nets nowadays (mostly, around the new-gen robotics policies) and use vast.ai instances with 8x RTX 4090 cards. I've been interested to give 8x MI300x a try, as they are supposed to be cheaper per FLOPs, but it looks like your service does not provide on-demand pay-per-second instances. Any plans to change that?
I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. It kind of makes sense since their…
> I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand.
Thank you for the response!
Renting 8 GPUs at once is fine and desired; it's not the issue.
The issue is that right now one has to commit to at least 1 week of use; my use patterns are bursty and it does not map well to the current proposition.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#48Earlier quoted context omitted.
I no longer have the 150,000 GPUs I was mining with. The only ones left standing are the small ones and most of them are just using what they already purchased. There is no liquidity in the shitcoins to be able to support daily large dumps of their tokens and unless you have almost free power, there is zero profitability.
You need to start a blog latchkey, I see you on here in all sorts of different contexts and it’s always interesting to see what you’re up to these days!
Thanks luke! Super appreciate it. Company website was the first priority, but I've also started on a blog component for it as well. Using bullet.so/notion.so and really enjoying how easy it is to build everything.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#49Earlier quoted context omitted.
I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. It kind of makes sense since their…
> I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. Thank you for the response! Renti…
This is very good feedback. We are just getting off the ground and on/off-boarding is still a bit of work for us.
Right now, we are trying to attract people who are a mix of wanting to kick the tires on a new product, as well as take compute for longer term.
I did mention in the pricing section that we can store your data locally, as part of the advertised pricing. This is our effort to recognize your use case.
I also understand that you want to optimize and don't want to pay for something that you're not using. We will eventually get to that point, but honestly just not there yet.
Think also from our end, we have these GPUs and if you're not using them... then who is? We've put out the capex/opex to make these available to you at any time, so the only way to be efficient on our side is to do a week long block right now.
Regardless, if you want to reach out to me directly, please do so. Maybe there is a middle ground we can both work from. Happy to consider all options and getting in early with us will always have first mover advantages.
Re: AMD Instinct MI325X in Q4 2024, 288GB of HBM3E
#50Earlier quoted context omitted.
I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. It kind of makes sense since their…
>I updated our pricing page to note this Nice job on your website! Sincerely, Someone who recently gave you a hard time for not having info on your website.
Thank you! That is awesome feedback. Your hard time helped motivate me to work on it as more of a priority.