Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

161–170 of 234 posts

Re: Ollama now supports AMD graphics cards

#161

Post by kergonath »

jameshart wrote: post #135
nerdix wrote: post #94

Earlier quoted context omitted.

The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

Compared to “ollama pull mixtral”? And then actually using the thing is easier as well.

Re: Ollama now supports AMD graphics cards

#162

Post by ies7 »

jameshart wrote: post #135
nerdix wrote: post #94

Earlier quoted context omitted.

The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

For us this may like a walk in the park.

For non technical people there is a possibility their os don't have git, wget and c++ compiler (especially in windows)

This is just like dropbox case years ago.

Re: Ollama now supports AMD graphics cards

#163

Post by refulgentis »

jart wrote: post #90
observationist wrote: post #88

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…

ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had…

Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt.

And they could have very, very, easily, there's a server, sitting right there.

They chose not to.

That's fine. But it's a choice.

The rest I chalk it up to inexperience and being busy.

Re: Ollama now supports AMD graphics cards

#164

Post by kergonath »

Karrot_Kream wrote: post #120
jart wrote: post #113

Earlier quoted context omitted.

Well you would be one of the few hundred people on the planet doing that. With local LLMs we're just trying to create a way for everyone else to use AI that doesn't require sharing all their data with them. First thing everyone asks for of course is how to turn the open source local llms into their own online service.

Ollama's purpose and usefulness is clear. I don't think anyone is disputing that nor the large usability gains ollama has driven. At least I'm not. As far as being one of the few hundred on the planet, well yeah that's why I'm on HN. There's tons of publications and subreddits and fora for generic tech conversation. I come here because I want to talk about the unknowns.

> I come here because I want to talk about the unknowns.

Your knowns an are unknowns to some people and vice versa. This is a great strength of HN; on a whole lot of subject you’ll find people ranging from enthusiastic to expert. There are probably subreddits or discord servers tailored to narrow niches and that’s cool, but HN is not that. They are complementary, if anything. In contrast, HN is much more interesting and with a much better S/N ratio than generic tech subreddits, it’s not even comparable.

Re: Ollama now supports AMD graphics cards

#165

Post by thejohnconway »

crooked-v wrote: post #82
eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

> Granted that it makes the experimentation process super easy That's the answer to your question. It may have less space than a Zune, but the average person doesn't care about technically superior alternatives that are much harder to use.

*Nomad

And lame.

Re: Ollama now supports AMD graphics cards

#166

Post by jameshart »

verdverm wrote: post #159
jameshart wrote: post #156

Earlier quoted context omitted.

Sure. Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.

I'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?

Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.

Re: Ollama now supports AMD graphics cards

#167

Post by Karrot_Kream »

kergonath wrote: post #164
Karrot_Kream wrote: post #120

Earlier quoted context omitted.

Ollama's purpose and usefulness is clear. I don't think anyone is disputing that nor the large usability gains ollama has driven. At least I'm not. As far as being one of the few hundred on the planet, well yeah that's why I'm on HN. There's tons of publications and subreddits and fora for generic tech conversation. I come here because I want to talk about the unknowns.

> I come here because I want to talk about the unknowns. Your knowns an are unknowns to some people and vice versa. This is a great strength of HN; on a whole lot of subject you’ll find people ranging from enthusiastic to expert. There are probably subreddits or discord servers tailored to narrow niches and that’s cool, but HN is not that. They are complementary, if anything. In contrast, HN is much more interesting…

I've been using this site since 2008. This is my second account from 2009. HN very much used to be a tech niche site. I realize that for newer users to HN, the appeal is like r/programming or r/technology but with a more text oriented interface or higher SNR or whatever but this is a shift in audience and there are folks like I on this site who still want to use it for niche content.

There are still threads where people do discuss gory details, even if the topics aren't technical. A lot of the mapping articles on the site bring out folks with deep knowledge about mapping stacks. Alternate energy threads do it too. It can be like that for LLMs also, but the user base has to want this site to be more than just Yet Another Tech News Aggregator thread.

For me as of late I've come to realize that the current audience wants YATNE more than they want deep discussion here and so I modulate my time here accordingly. The LLM threads bring me here because experts like jart chime in.

Re: Ollama now supports AMD graphics cards

#168

Post by verdverm »

jameshart wrote: post #166
verdverm wrote: post #159

Earlier quoted context omitted.

I'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?

Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifi…

I've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask

Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs

Re: Ollama now supports AMD graphics cards

#169

Post by kergonath »

Karrot_Kream wrote: post #167
kergonath wrote: post #164

Earlier quoted context omitted.

> I come here because I want to talk about the unknowns. Your knowns an are unknowns to some people and vice versa. This is a great strength of HN; on a whole lot of subject you’ll find people ranging from enthusiastic to expert. There are probably subreddits or discord servers tailored to narrow niches and that’s cool, but HN is not that. They are complementary, if anything. In contrast, HN is much more interesting…

I've been using this site since 2008. This is my second account from 2009. HN very much used to be a tech niche site. I realize that for newer users to HN, the appeal is like r/programming or r/technology but with a more text oriented interface or higher SNR or whatever but this is a shift in audience and there are folks like I on this site who still want to use it for niche content. There are still threads where peo…

> I've been using this site since 2008. This is my second account from 2009.

I did not really like HN back in the day because it felt too startup-y, but maybe I got a wrong impression. I much preferred Ars Technica and their forum (now, Ars is much less compelling).

> For me as of late I've come to realize that the current audience wants YATNE more than they want deep discussion here and so I modulate my time here accordingly.

I think it depends on the stories. Different subjects have different demographics, and I have almost completely stopped reading physics stuff because it is way too Reddit-like (and full of confident people asserting embarrassingly wrong facts). I can see how you could feel about fields closer to your interests being dumbed down by over-confident non-specialists.

There are still good, highly technical discussions, but it is true that the home page is a bit limited and inefficient to find them.

Re: Ollama now supports AMD graphics cards

#170

Post by [deleted] »

Karrot_Kream wrote: post #167
kergonath wrote: post #164

Earlier quoted context omitted.

> I come here because I want to talk about the unknowns. Your knowns an are unknowns to some people and vice versa. This is a great strength of HN; on a whole lot of subject you’ll find people ranging from enthusiastic to expert. There are probably subreddits or discord servers tailored to narrow niches and that’s cool, but HN is not that. They are complementary, if anything. In contrast, HN is much more interesting…

I've been using this site since 2008. This is my second account from 2009. HN very much used to be a tech niche site. I realize that for newer users to HN, the appeal is like r/programming or r/technology but with a more text oriented interface or higher SNR or whatever but this is a shift in audience and there are folks like I on this site who still want to use it for niche content. There are still threads where peo…

[deleted]
Post reply on HN