Ollama now supports AMD graphics cards • Hacker News Board

Re: Ollama now supports AMD graphics cards

#141

Post by verdverm » Fri, Mar 15, 2024, 11:46 PM UTC

jameshart wrote: post #135

nerdix wrote: post #94

Earlier quoted context omitted.

The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

This shows the value ollama provides

I only need to know the model name and then run a single command

Re: Ollama now supports AMD graphics cards

#142

Post by eclectic29 » Fri, Mar 15, 2024, 11:46 PM UTC

jart wrote: post #113

Karrot_Kream wrote: post #108

Earlier quoted context omitted.

I mean, it takes something difficult like an LLM and makes it easy to run. It's bound to get attention. If you've tried to get other models like BERT based models to run you'll realize just how big the usability gains are running ollama than anything else in the space. If the question you're asking is why so many folks are focused on experimentation instead of productionizing these models, then I see where you're com…

Well you would be one of the few hundred people on the planet doing that. With local LLMs we're just trying to create a way for everyone else to use AI that doesn't require sharing all their data with them. First thing everyone asks for of course is how to turn the open source local llms into their own online service.

Few hundred on the planet? Are you kidding me? We're asking enterprises to run LLMs on-premise (I'm intentionally discounting the cloud scenario where the traffic rates are much higher). That's way more than a hundred and sorry to break it to you that Ollama is just not going to cut it.

Re: Ollama now supports AMD graphics cards

#143

Post by eclectic29 » Fri, Mar 15, 2024, 11:49 PM UTC

airocker wrote: post #104

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

More than one is easy: put it behind a load balancer. Put one ollama in one container or one port.

FWIW Ollama has no concurrency support even though llama.cpp's server component (the thing that Ollama actually uses) supports it. Besides, you can't have more than 1 model running. Unloading and loading models is not free. Again, there's a lot more and really much of the real optimization work is not in Ollama; it's in llama.cpp which is completely ignored in this equation.

Re: Ollama now supports AMD graphics cards

#144

Post by verdverm » Fri, Mar 15, 2024, 11:50 PM UTC

haolez wrote: post #121

Is there an equivalent to ollama or gpt4all for Android? I'd like to host my model somewhere and talk to it via app.

At that point, it sounds like your after an API endpoint for any model. Lots of solutions out there, depends more on your hosting

Re: Ollama now supports AMD graphics cards

#145

Post by eclectic29 » Fri, Mar 15, 2024, 11:50 PM UTC

airocker wrote: post #132

Zambyte wrote: post #128

Earlier quoted context omitted.

That is still one model per instance of Ollama, right?

yes, not sure you can do better than that. You cannot still have one instance of LLM in (GPU) memory answer two queries at one time.

Of course, you can support concurrent requests. But Ollama doesn't support it and it's not meant for this purpose and that's perfectly ok. That's not the point though. For fast/perf scenarios, you're better off with vllm.

Re: Ollama now supports AMD graphics cards

#146

Post by reissbaker » Fri, Mar 15, 2024, 11:51 PM UTC

jon_richards wrote: post #112

ronsor wrote: post #103

Earlier quoted context omitted.

Realistically that's the only way to reduce the amount of bugs in your C++ code.

Do you want machine spirits? Because that's how you get machine spirits.

My brother in Stroustrup, we are trying to run llama.cpp, we are definitionally attempting to summon machine spirits.

Re: Ollama now supports AMD graphics cards

#147

Post by Karrot_Kream » Fri, Mar 15, 2024, 11:51 PM UTC

eclectic29 wrote: post #142

jart wrote: post #113

Earlier quoted context omitted.

Well you would be one of the few hundred people on the planet doing that. With local LLMs we're just trying to create a way for everyone else to use AI that doesn't require sharing all their data with them. First thing everyone asks for of course is how to turn the open source local llms into their own online service.

Few hundred on the planet? Are you kidding me? We're asking enterprises to run LLMs on-premise (I'm intentionally discounting the cloud scenario where the traffic rates are much higher). That's way more than a hundred and sorry to break it to you that Ollama is just not going to cut it.

No need to be angry about this. Tech folks should be discussing this collectively and collaboratively. There's space for everything from local models running on smartphones all the way up to OpenAI style industrialized models. Back when social networks were first coming out, I used to read lots of comments about deploying and running distributed systems. I remember reading early incident reports about hotspotting and consistent hashing and TTL problems and such. We need to foster more of that kind of conversation for LMs. Sadly right now Xitter seems to be the best place for that.

Re: Ollama now supports AMD graphics cards

#148

Post by eclectic29 » Fri, Mar 15, 2024, 11:52 PM UTC

verdverm wrote: post #141

jameshart wrote: post #135

Earlier quoted context omitted.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

This shows the value ollama provides I only need to know the model name and then run a single command

And what will you do after trying it? Sure, you saved a few mins in trying out a model or models. What next?

Re: Ollama now supports AMD graphics cards

#149

Post by ecnahc515 » Fri, Mar 15, 2024, 11:53 PM UTC

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

Ollama is the Docker of LLMs. Ollama made it _very_ easy to run LLMs locally. This is surprisingly not as easy as it seems, and incredibly useful.

Re: Ollama now supports AMD graphics cards

#150

Post by airocker » Fri, Mar 15, 2024, 11:53 PM UTC

eclectic29 wrote: post #145

airocker wrote: post #132

Earlier quoted context omitted.

yes, not sure you can do better than that. You cannot still have one instance of LLM in (GPU) memory answer two queries at one time.

Of course, you can support concurrent requests. But Ollama doesn't support it and it's not meant for this purpose and that's perfectly ok. That's not the point though. For fast/perf scenarios, you're better off with vllm.

Thanks! This is great to know.