Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

151–160 of 234 posts

Re: Ollama now supports AMD graphics cards

#151

Post by verdverm »

eclectic29 wrote: post #148
verdverm wrote: post #141

Earlier quoted context omitted.

This shows the value ollama provides I only need to know the model name and then run a single command

And what will you do after trying it? Sure, you saved a few mins in trying out a model or models. What next?

I focus on building the application rather than figuring out someone else preferred method for how I should work?

I use Docker Compose locally, Kubernetes in the cloud

I run in hot-reload locally, I build for production

I often nuke my database locally, but I run it HA in production

It is very rare to use the same technology locally (or the same way) as in production

Re: Ollama now supports AMD graphics cards

#152

Post by jameshart »

verdverm wrote: post #141
jameshart wrote: post #135

Earlier quoted context omitted.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

This shows the value ollama provides I only need to know the model name and then run a single command

It should be fairly obvious that one can find alternative models and use them in the above command too.

Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery.

Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. It’s just something you can do by running code that loads a model file into memory and feeds tokens to it.

More programmers should be looking at llama.cpp language bindings than at Ollama’s implementation of the openAI api.

Re: Ollama now supports AMD graphics cards

#153

Post by airocker »

eclectic29 wrote: post #143
airocker wrote: post #104

Earlier quoted context omitted.

More than one is easy: put it behind a load balancer. Put one ollama in one container or one port.

FWIW Ollama has no concurrency support even though llama.cpp's server component (the thing that Ollama actually uses) supports it. Besides, you can't have more than 1 model running. Unloading and loading models is not free. Again, there's a lot more and really much of the real optimization work is not in Ollama; it's in llama.cpp which is completely ignored in this equation.

Thanks! Great to know. I did not know llama.cpp could do this. It should be pretty straight forward to support, not sure why they would not do it.

Re: Ollama now supports AMD graphics cards

#154

Post by renewiltord »

segmondy wrote: post #136
renewiltord wrote: post #24

Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.

You don't need 128 lanes. 8x PCIe3 is more than enough, so for 4 cards that's 32. Most CPUs have about 40lanes. If you are not doing much that would be more than sufficient. Buy a PCIe riser. Go to amazon and search for it, a 16x to 16x PCIe riser. They go for about $25-$30 often about 20-30cm. If you want really long one, you can get one from China a 60cm for about the same price, you just have to wait for 3 weeks.…

Ah ha, that's the part I was curious about. I was wondering if I could keep everything cool open rig. I'm waiting for the stuff to arrive: risers, board, CPU, GPUs. And I've been putting it off because I wasn't sure how about the case. All right then, open rig frame. Thank you!

Re: Ollama now supports AMD graphics cards

#155

Post by verdverm »

jameshart wrote: post #152
verdverm wrote: post #141

Earlier quoted context omitted.

This shows the value ollama provides I only need to know the model name and then run a single command

It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. I…

I'd rather focus on building on top of of LLMs than going lower level

Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box

Re: Ollama now supports AMD graphics cards

#156

Post by jameshart »

verdverm wrote: post #155
jameshart wrote: post #152

Earlier quoted context omitted.

It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. I…

I'd rather focus on building on top of of LLMs than going lower level Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box

Sure.

Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.

Re: Ollama now supports AMD graphics cards

#157

Post by eclectic29 »

Karrot_Kream wrote: post #147
eclectic29 wrote: post #142

Earlier quoted context omitted.

Few hundred on the planet? Are you kidding me? We're asking enterprises to run LLMs on-premise (I'm intentionally discounting the cloud scenario where the traffic rates are much higher). That's way more than a hundred and sorry to break it to you that Ollama is just not going to cut it.

No need to be angry about this. Tech folks should be discussing this collectively and collaboratively. There's space for everything from local models running on smartphones all the way up to OpenAI style industrialized models. Back when social networks were first coming out, I used to read lots of comments about deploying and running distributed systems. I remember reading early incident reports about hotspotting and…

Not angry. Having a discussion :-). It just amazes me how the HN crowd is more than happy with just trying out a model on their machine and calling it a day and not seeing the real picture ahead. Let ignore perf concerns for a moment. Let's say I want to run it on a shared server in the enterprise network so that any application can make use of it. Each application might want to use a model of their choosing. Ollama will unload/load/unload models as each new request arrives. Not sure if folks here are realizing this :-)

Re: Ollama now supports AMD graphics cards

#158

Post by mkarrmann »

dheera wrote: post #101
observationist wrote: post #88

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…

[flagged]

I've recently taken to calling it llama.cpp plus ollama

Re: Ollama now supports AMD graphics cards

#159

Post by verdverm »

jameshart wrote: post #156
verdverm wrote: post #155

Earlier quoted context omitted.

I'd rather focus on building on top of of LLMs than going lower level Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box

Sure. Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.

I'm aware, I don't need that amount of sophistication yet.

Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?

Re: Ollama now supports AMD graphics cards

#160

Post by kergonath »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

> It's not meant for production deployments.

I am probably not the demographics you expect. I don’t do “production” in that sense, but I have ollama running quite often when I am working, as I use it for RAG and as a fancy knowledge extraction engine. It is incredibly useful:

- I can test a lot of models by just pulling them (very useful as progress is very fast),

- using their command line is trivial,

- the fact that it keeps running in the background means that it starts once every few days and stays out of the way,

- it integrates nicely with langchain (and a host of other libraries), which means that it is easy to set up some sophisticated process and abstract away the LLM itself.

> what do you do after the initial experimentation?

I just keep using it. And for now, I keep tweaking my scripts but I expect them to stabilise at some point, because I use these models to do some real work, and this work is not monkeying about with LLMs.

> I'm not even talking about giving credit to llama.cpp, just mentioning that this product is gaining disproportionate attention and kudos compared to the value it delivers.

For me, there is nothing that comes close in terms of integration and convenience. The value it delivers is great, because it enables me to do some useful work without wasting time worrying about lower-level architecture details. Again, I am probably not in the demographics you have in mind (I am not a CS person and my programming is usually limited to HPC), but ollama is very useful to me. Its reputation is completely deserved, as far as I am concerned.

Post reply on HN