AMD Instinct MI325X in Q4 2024, 288GB of HBM3E

Post by krasin » Tue, Jun 04, 2024, 4:56 AM UTC

latchkey wrote: post #49

krasin wrote: post #47

Earlier quoted context omitted.

> I would love nothing more than to be able to enable on-demand GPUs, but unfortunately this is a limitation from AMD right now. We can't do PCIe pass through to a virtual machine, it just doesn't work. This is why our minimum is 8 right now. If you look at all of our competitors, they have the same issue. Even Azure "VM", is 8 at a time, but they are all sold out due to high demand. Thank you for the response! Renti…

This is very good feedback. We are just getting off the ground and on/off-boarding is still a bit of work for us. Right now, we are trying to attract people who are a mix of wanting to kick the tires on a new product, as well as take compute for longer term. I did mention in the pricing section that we can store your data locally, as part of the advertised pricing. This is our effort to recognize your use case. I als…

Post by Netcob » Tue, Jun 04, 2024, 7:58 AM UTC

SushiHippie wrote: post #39

Netcob wrote: post #36

Earlier quoted context omitted.

In AI, that doesn't sound too surprising to me right now. I just experiment with some local LLMs, but the differences are pretty huge: Llama 3 8B, Raspberry Pi 5: 2-3 Tokens/second (but it works!) Llama 3 8B, RTX 4080: ~60 Tokens/second Llama 3 8B, groq.com LPU, ~1300 Tokens/second Llama 3 70B, AMD 7800X3D: 1-2 Tokens/second Llama 3 70B, groq.com LPU, ~330 Tokens/second There seem to be huge gaps between CPU, GPU and…

What quantization levels did you use? I think groq doesn't use quantization, so the gap between your hardware and groq would be even further apart.