CUDA Moat Still Alive
semianalysis.com
CUDA Moat Still Alive
1–10 of 176 posts
Re: CUDA Moat Still Alive
#2What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers.
I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
Re: CUDA Moat Still Alive
#3What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.
Re: CUDA Moat Still Alive
#4What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.
Re: CUDA Moat Still Alive
#5> It’s not just that it’s immature software, they need to change how they do development.
I remember geohot saying something similar about a year ago
Re: CUDA Moat Still Alive
#6B100, B200 ramp-up is only 4-6 months away.
Re: CUDA Moat Still Alive
#7What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.
Re: CUDA Moat Still Alive
#8Sounds like a buy signal for AMD. If you run the right branch and set the right env cars, the thing flies.
Re: CUDA Moat Still Alive
#9Sounds like a buy signal for AMD. If you run the right branch and set the right env cars, the thing flies.
Unfortunately,
> Getting reasonable training performance out of AMD MI300X is an NP-Hard problem.
Re: CUDA Moat Still Alive
#10What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows ( https://rocm.docs.amd.com/projects/install-on-windows/en/lat... ) and not Linux ( https://rocm.docs.amd.com/projects/install-on-linux/en/lates... ), where most AI training and inference occurs.…
In practice, consumer Navi 21 based cards (RX 6900XT etc) and Navi 31 cards (RX 7900 XTX etc) are compatible with Pytorch on Linux.
What they write about ROCm and Windows is equivocation. They target only one app: Blender. Pytorch+ROCm+Windows does not work.
I had bought a 6900XT myself around launch time (the RTX3080 I ordered was not coming, it was the chip shortage times...) and it took around 2 years for Pytorch to become actually usable on it.