Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

191–200 of 234 posts

Re: Ollama now supports AMD graphics cards

#191

Post by 65a »

rahimnathwani wrote: post #188
65a wrote: post #185

Earlier quoted context omitted.

Yep, and it deserves the credit! He who writes the cuda kernel (or translates it) controls the spice. I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814 . I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.

I'm curious about both: - what's special about the memory allocation, and how might it help me? - what are you now using instead of ollama?

Ollama does a nice job of looking at how much VRAM the card has and tuning the number of gpu layers offloaded. Before that, I mainly just had to guess. It's still a heuristic, but I thought that was neat.

I'm mainly just using llama.cpp as a native library now, mainly for the direct access to more of llama's data structures, and because I have a sort of unique sampler setup.

Re: Ollama now supports AMD graphics cards

#192

Post by ramblerman »

eclectic29 wrote: post #148
verdverm wrote: post #141

Earlier quoted context omitted.

This shows the value ollama provides I only need to know the model name and then run a single command

And what will you do after trying it? Sure, you saved a few mins in trying out a model or models. What next?

Relax. Not everything in this world was built exactly for you. You almost seem to have a problem with this.

Re: Ollama now supports AMD graphics cards

#193

Post by rkwz »

kergonath wrote: post #160
eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

> It's not meant for production deployments. I am probably not the demographics you expect. I don’t do “production” in that sense, but I have ollama running quite often when I am working, as I use it for RAG and as a fancy knowledge extraction engine. It is incredibly useful: - I can test a lot of models by just pulling them (very useful as progress is very fast), - using their command line is trivial, - the fact tha…

> I use it for RAG and as a fancy knowledge extraction engine

Curious, can you share more details about your usecase?

Re: Ollama now supports AMD graphics cards

#194

Post by varjag »

refulgentis wrote: post #163
jart wrote: post #90

Earlier quoted context omitted.

ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had…

Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt. And they could have very, very, easily, there's a server, sitting right there. They chose not to. That's fine. But it's a choice. The rest I chalk it up to inexperience and being busy.

Where the name does come from?

Re: Ollama now supports AMD graphics cards

#195

Post by kleiba »

refulgentis wrote: post #163
jart wrote: post #90

Earlier quoted context omitted.

ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had…

Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt. And they could have very, very, easily, there's a server, sitting right there. They chose not to. That's fine. But it's a choice. The rest I chalk it up to inexperience and being busy.

OP never said "contributions to llama.cpp", the comment just said just "contributions" which I read as "own developments".

Re: Ollama now supports AMD graphics cards

#196

Post by icelain »

nerdix wrote: post #94
eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

Check out EasyDiffusion.

Re: Ollama now supports AMD graphics cards

#197

Post by VadimPR »

Ollama runs really, really slow on my MBP for Mistral - as in just a few tokens a second and it takes a long while before it starts giving a result. Anyone else run into this?

I've seen that it seems to be related to the amount of system memory available when ollama is started (??) however LM Studio does not have such issues.

Re: Ollama now supports AMD graphics cards

#198

Post by [deleted] »

observationist wrote: post #88

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…

[deleted]

Re: Ollama now supports AMD graphics cards

#199

Post by karolist »

wongarsu wrote: post #49
choilive wrote: post #15

Earlier quoted context omitted.

They bump up VRAM because they can't compete on raw compute.

Or rather Nvidia is purposefully restricting VRAM to avoid gaming cards canibalizing their supremely profitable professional/server cards. AMD has no relevant server cards, so they have no reason to hold back on VRAM in consumer cards

Nvidia released consumer RTX 3090 with 24GB VRAM in Sep 2020, AMDs flagship release in that same month was 6900 XT with 16GB VRAM. Who is being restrictive here exactly?

Re: Ollama now supports AMD graphics cards

#200

Post by read_if_gay_ »

verdverm wrote: post #141
jameshart wrote: post #135

Earlier quoted context omitted.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

This shows the value ollama provides I only need to know the model name and then run a single command

>Hacker News
Post reply on HN