No support for my rx 5700xt :(
Ollama now supports AMD graphics cards
131–140 of 234 posts
Re: Ollama now supports AMD graphics cards
#132Earlier quoted context omitted.
More than one is easy: put it behind a load balancer. Put one ollama in one container or one port.
That is still one model per instance of Ollama, right?
yes, not sure you can do better than that. You cannot still have one instance of LLM in (GPU) memory answer two queries at one time.
Re: Ollama now supports AMD graphics cards
#133Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.
My 1080ti runs ok even this 47B model: https://ollama.com/library/dolphin-mixtral
Re: Ollama now supports AMD graphics cards
#134Hm, fooocus manages to run, but for Ollama I get: >time=2024-03-16T00:11:07.993+01:00 level=WARN source=amd_linux.go:50 msg="ollama >recommends running the https://www.amd.com/en/support/linux-drivers : amdgpu version file >missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or >directory" >time=2024-03-16T00:11:07.993+01:00 level=INFO source=amd_linux.go:85 msg="detected amdgpu >version…
Ah, this is probably from missing ROCm libraries. The dynamic libraries are available as one of the release assets (warning: it's about 4GB expanded) https://github.com/ollama/ollama/releases/tag/v0.1.29 – dropping them in the same directory as the `ollama` binary should work.
Re: Ollama now supports AMD graphics cards
#135I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like:
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make
wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true
./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128Re: Ollama now supports AMD graphics cards
#136Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.
You don't need 128 lanes. 8x PCIe3 is more than enough, so for 4 cards that's 32. Most CPUs have about 40lanes. If you are not doing much that would be more than sufficient. Buy a PCIe riser. Go to amazon and search for it, a 16x to 16x PCIe riser. They go for about $25-$30 often about 20-30cm. If you want really long one, you can get one from China a 60cm for about the same price, you just have to wait for 3 weeks. That's what I did. Stuffing all those in a case is often difficult, so you have to go open rig. Either have the cables running out your computer and figuring out a way to support the cards while keeping them cool or just buy a small $20-$30 open rig frame.
Re: Ollama now supports AMD graphics cards
#137I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
I am 100% uninterested in your production deployment of rent seeking behavior for tools and models I can run myself. Ollama empowers me to do more of that easier. That’s why it’s popular.
OP’s point is more that Ollama isn’t what’s doing the empowering. Llama.cpp is.
Re: Ollama now supports AMD graphics cards
#138Earlier quoted context omitted.
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128
Last time I tried llama.cpp I got errors when running make that were way too time consuming to bother tracking down.
It's probably a simple build if everything is how it wants it, but it wasn't in my machine, while running ollama was.
Re: Ollama now supports AMD graphics cards
#139There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…
[flagged]
Trust me, they do. Programmers curse him out on a daily basis.
Re: Ollama now supports AMD graphics cards
#140I heard "Nvidia for LLM today is similar to how Sun Microsystems was for the web"
... for a very brief period of time until Linux servers and other options caught up.
OS was possibly much more complicated to write at that time than CUDA is to write today. And competition is too strong. It might be more briefer than even Sun.