Is there an equivalent to ollama or gpt4all for Android? I'd like to host my model somewhere and talk to it via app.
Ollama now supports AMD graphics cards
121–130 of 234 posts
Re: Ollama now supports AMD graphics cards
#122I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
On a mac, https://drawthings.ai is the ollama of Stable Diffusion.
Re: Ollama now supports AMD graphics cards
#123It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support. Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that the…
While the PRs went in slightly earlier, much of the time was spent on testing the integrations, and working with AMD directly to resolve issues. There were issues that we resolved prior to cutting the release, and many reported by the community as well.
Thank you for clarifying and thanks for the great work you do!
Re: Ollama now supports AMD graphics cards
#124Earlier quoted context omitted.
Realistically that's the only way to reduce the amount of bugs in your C++ code.
Do you want machine spirits? Because that's how you get machine spirits.
lol, although you made me think--the history of computers has involved patching layer after layer of sediment on top of each other until the how things work 10 layers deep is forgotten. Imagine when local LLMs are five layers deep. It'll be like having machine spirits. You know how you have to yell to get the LLM to do what you want sometimes? That's what the future version of sudo will be like.
Re: Ollama now supports AMD graphics cards
#125Any info on the performance? How does it compare to 4080/4090?
Re: Ollama now supports AMD graphics cards
#126I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
For me, ComfyUI made the process of installing and playing with SD about as simple as a Windows installer.
Re: Ollama now supports AMD graphics cards
#127Hm, fooocus manages to run, but for Ollama I get:
>time=2024-03-16T00:11:07.993+01:00 level=WARN source=amd_linux.go:50 msg="ollama >recommends running the https://www.amd.com/en/support/linux-drivers: amdgpu version file >missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or >directory" >time=2024-03-16T00:11:07.993+01:00 level=INFO source=amd_linux.go:85 msg="detected amdgpu >versions [gfx1031]" >time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:339 msg="amdgpu >detected, but no compatible rocm library found. Either install rocm v6, or follow manual >install instructions at install" rel="nofollow">https://github.com/ollama/ollama/blob/main/docs/linux.md#man..." >time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:96 msg="unable to verify >rocm library, will use cpu: no suitable rocm found, falling back to CPU" >time=2024-03-16T00:11:07.996+01:00 level=INFO source=routes.go:1105 msg="no GPU detected"
Need to check how to install rocm on arch again... have done it once, a few moons back, but alas...
Re: Ollama now supports AMD graphics cards
#128I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
More than one is easy: put it behind a load balancer. Put one ollama in one container or one port.
That is still one model per instance of Ollama, right?
Re: Ollama now supports AMD graphics cards
#129Earlier quoted context omitted.
Not sure why you're downvoted, but as far as I've heard AMD cards can't beat 4090 - yet. Still, I think AMD will catch or overtake NVidia in hardware soon, but software is a bigger problem. Hopefully the opensource strategy will pay off for them.
A RTX 4090 is about twice the price of and 50%-ish faster than AMD's most expensive consumer card so I'm not sure anyone really expects it to ever surpass a 4090. A 7900 XTX beating a RTX 4080 at inference is probably a more realistic goal though I'm not sure how they compare right now.
The 4080 is $1k for 16gb of VRAM, and the 7900 is $1k for 24gb of VRAM. Unless you're constantly hammering it with requests, the extra speed you may get with CUDA on a 4080 is basically irrelevant when you can run much better models at a reasonable speed.
Re: Ollama now supports AMD graphics cards
#130[deleted]