Given the price of top line NVidia cards, if they can be had at all, there's got to be a lot of effort going on behind the scenes to improve AMD support in various places.
What are the barriers to doing so?
Ollama now supports AMD graphics cards
181–190 of 234 posts
Re: Ollama now supports AMD graphics cards
#182It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support. Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that the…
Maybe they wanted to wait for bug reports for 21 days before publishing a popular blog post about it..?
Re: Ollama now supports AMD graphics cards
#183Earlier quoted context omitted.
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128
This will likely build a version without GPU acceleration, I think?
Re: Ollama now supports AMD graphics cards
#184Earlier quoted context omitted.
Used crypto mining parts not available any more?
Crypto mining didn't require significant bandwidth to the card. Mining-oriented motherboards typically only provisioned a single lane of PCIe to each card, and often used anemic host CPUs (like Celeron embedded parts).
Does LLM inference require significant bandwidth to the card? You have to get the model into VRAM, but that's a fixed startup cost, not a per-output-token cost.
Re: Ollama now supports AMD graphics cards
#185Here is the commit that added ROCm support to llama.cpp back in August: https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
Yep, and it deserves the credit! He who writes the cuda kernel (or translates it) controls the spice.
I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814. I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.
Re: Ollama now supports AMD graphics cards
#186Earlier quoted context omitted.
ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had…
Newton only gave credits to "Gods". He said he stands on shoulders of Gods or something like that. But he never mentioned which Gods in particular, did he?
Giants, not Gods, and it's debated whether it was actually an insult or not: https://en.wikipedia.org/wiki/Isaac_Newton#Personality
Re: Ollama now supports AMD graphics cards
#187I'm thrilled to see support for RX 6800/6800 XT / 6900 XT. I bought one of those for an outrageous amount during the post-covid shortage in hopes that I could use it for ML stuff, and thus far it hasn't been very successful, which is a shame because it's a beast of a card! Many thanks to ollama project and llama.cpp!
Sad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.
I've been using my 6750XT for more than a year now on all sorts of AI projects. Takes a little research and a few env vars but no need to wait for "official" support most of the time.
Re: Ollama now supports AMD graphics cards
#188Here is the commit that added ROCm support to llama.cpp back in August: https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
Yep, and it deserves the credit! He who writes the cuda kernel (or translates it) controls the spice. I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814 . I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.
I'm curious about both:
- what's special about the memory allocation, and how might it help me?
- what are you now using instead of ollama?
Re: Ollama now supports AMD graphics cards
#189Does it also support AMD APUs?
Re: Ollama now supports AMD graphics cards
#190Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.
You don't need 128 lanes. 8x PCIe3 is more than enough, so for 4 cards that's 32. Most CPUs have about 40lanes. If you are not doing much that would be more than sufficient. Buy a PCIe riser. Go to amazon and search for it, a 16x to 16x PCIe riser. They go for about $25-$30 often about 20-30cm. If you want really long one, you can get one from China a 60cm for about the same price, you just have to wait for 3 weeks.…
Most CPUs have about 20 lanes (plus a link to the chipset).
On the one hand, they will be gen 4 or 5, so they're the equivalent of 40-80 gen 3 lanes.
On the other hand, you can only split them up if you have a motherboard that supports bifurcation. If you buy the wrong model, you're stuck dedicating the equivalent of 64 gen 3 lanes to a single card.
Edit: Actually, looking into it further, current Intel desktop processors will only run their lanes as 16(+4) or 8+8(+4). You can kind of make 4 cards work by using chipset-fed slots, but that sucks. You could also get a PCIe switch but those are very expensive. AMD will do 4+4+4+4(+4) on the right boards.