Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

221–230 of 234 posts

Re: Ollama now supports AMD graphics cards

#221

Post by jart »

imtringued wrote: post #211
jart wrote: post #117

Earlier quoted context omitted.

I've heard geohot is upset about that. I haven't tortured any of my AMD cards enough to run into that issue yet. Do you know how to make it happen?

Last time I used AMD GPUs for GPGPU all it took was running hashcat to make the desktop rendering unstable. I'm sure leaving it run overnight would've gotten me a system crash.

That's always happened with NVIDIA on Linux too, because Linux is an operating system that actually gives you the resources you ask for. Consider using a separate video card that's dedicated to your video needs. Otherwise you should use MacOS or Windows. It's 10x slower at building code. But I can fork bomb it while training a model and Netflix won't skip a frame. Yes I've actually done this.

Re: Ollama now supports AMD graphics cards

#222

Post by refulgentis »

jart wrote: post #220
refulgentis wrote: post #218

Earlier quoted context omitted.

Fair. I didn't want to assume the worst, that it was just rhetorical slight of hand where "providing release packaging around an open source? you should credit it, at some point, somewhere." is implied as ridiculous, like asking llama.cpp to credit Google Brain. (Presumably, the implication is, for transformers / the Attention is All You Need paper)

If you're going to accuse me of rhetorical sleight of hand, you could start by at least spelling it correctly. This whole code stealing shtick is the kind of thing I'd expect from teenagers on 4chan not from someone who's been professionally trained like you. Many open source licenses like BSD-4 and X11 are actually written to prohibit people from "giving credit" in advertising in the manner you're expecting.

I'm 35, got my start in FOSS by working on Handbrake at 17 when ffmpeg was added.

There, I learned that you're supposed to credit projects you depend on, especially ones you depend on heavily.

I don't know why you keep finding ways to dismiss this simple fact. (really? spelling? on Saturday morning!?!? :D).

Especially with a strong record of open source contributions yourself.

Especially when your project is a classic example of A) building around llama.cpp and crediting it. I literally was thinking about llamafile when I wrote my original comment, before I realized who I was replying to.

I'm really trying to find a communication bridge here because I'm deeply curious, and I'd appreciate you doing the same if I'm lucky enough to get a reply from your august personage again. (seriously! no sarcasm!) My latest guesses:

- you saw this post far after the early tide of, ex., exaggerating for clarity, "anyone got the leak on the deets on how these wizards did this?!?!?! CUDA going down!"

- You're unaware Ollama _does not mention or credit llama.cpp at all_. Not once. Never. Google search query I used to verify my presumption is `site:ollama.com "llama.cpp"`. You will find that it is only mentioned in READMEs of repos of other peoples models, mentioning how they quantized.

- You're unaware this is an ongoing situation. Probably the 3rd thread I've seen in 3 months with decreasing #s of people treating it like a independent commercial startup making independent breakthroughs, and increasing #s of people being like "...why are you still doing this..."

For those unfamiliar, this is how jart's llamafile project credits llama.cpp, they certainly don't avoid it altogether, and they certainly don't seem to think its unnecessary. (source: https://github.com/Mozilla-Ocho/llamafile)

- 2nd sentence in README: "Our goal is to make open LLMs much more accessible to both developers and end users. We're doing that by combining ___llama.cpp___ with Cosmopolitan Libc into one framework"

- 21 mentions in README altogether.

- Under "How llamafile works", 3 mentions crediting llama.cpp in 5 steps.

- Announcement blog post: 4 mentions, Justine co-authored it. https://hacks.mozilla.org/2023/11/introducing-llamafile/

Re: Ollama now supports AMD graphics cards

#223

Post by jart »

refulgentis wrote: post #222
jart wrote: post #220

Earlier quoted context omitted.

If you're going to accuse me of rhetorical sleight of hand, you could start by at least spelling it correctly. This whole code stealing shtick is the kind of thing I'd expect from teenagers on 4chan not from someone who's been professionally trained like you. Many open source licenses like BSD-4 and X11 are actually written to prohibit people from "giving credit" in advertising in the manner you're expecting.

I'm 35, got my start in FOSS by working on Handbrake at 17 when ffmpeg was added. There, I learned that you're supposed to credit projects you depend on, especially ones you depend on heavily. I don't know why you keep finding ways to dismiss this simple fact. (really? spelling? on Saturday morning!?!? :D). Especially with a strong record of open source contributions yourself. Especially when your project is a classi…

Look, there's a very simple way we can prove or disprove if ollama is doing something wrong. The MIT license (which llama.cpp uses) requires that their copyright notice accompany their source code. Here we can see that ollama is using llama.cpp (there's no secret about that) but they don't have any llama.cpp source code in their repo apparently https://github.com/search?q=repo%3Aollama%2Follama+Georgi+Ge... since they appear to be building it as a static library separately as some kind of workflow in their CI system.

But here's the twist: the MIT license requires that the copyright notice be distributed with the binary forms as well. That does not mean advertising. They're not required to mention it in their website or in their communiqués. The bare minimum requirement is that the copyright notice be present in their release artifacts.

llamafile solves this by embedding the copyright notice inside your llamafiles.

    $ grep 'Georgi Gerganov' llamafile-0.6.2
    grep: llamafile-0.6.2: binary file matches
So that way you don't have to worry about giving credit and can continue having a nice single file binary. The code which does that is here: https://github.com/Mozilla-Ocho/llamafile/blob/9c53c272650bb...

So let me install the latest Ollama on my Windows computer and see if they're doing this too. https://justine.lol/tmp/ollama-license-violation.png It would seem the answer is no. So yes, ollama appears to be violating the llama.cpp license, and probably the licenses of many other projects too. But not for the reasons we were discussing earlier.

Let's try Linux.

    $ curl -fsSL https://ollama.com/install.sh | sh
    $ grep -R 'Georgi Gerganov' /usr/local/bin/ollama
    $?=1 $
Oops. They're violating the license on Linux too. It's also a little creepy that it used sudo on its own. So I've filed an issue here: https://github.com/ollama/ollama/issues/3185

Re: Ollama now supports AMD graphics cards

#224

Post by mejutoco »

Zambyte wrote: post #51
coconut08 wrote: post #34

Earlier quoted context omitted.

it doesn't matter how much compute you have if you don't have enough vram to run the model.

Exactly. My friend was telling me that I was making a mistake for getting a 7900 XTX to run language models, when the fact of the matter is the cheapest NVIDIA card with 24 GB of VRAM is over 50% more expensive than the 7900 XTX. Running a high quality model at like 80 tps is way more important to me than running a way lower quality model at like 120 tps.

I cannot parse this. The Radeon RX 7900 XTX also has 24GB of vram, so how does it help you run higher quality models? I would understand if it had more ram.

Re: Ollama now supports AMD graphics cards

#225

Post by kergonath »

kergonath wrote: post #213
rkwz wrote: post #193

Earlier quoted context omitted.

> I use it for RAG and as a fancy knowledge extraction engine Curious, can you share more details about your usecase?

The use case is exploratory literature review in a specific scientific field. I have a setup that takes pdfs and does some OCR and layout detection with Amazon, and then bunch them with some internal reports. Then, I have a pipeline to write summaries of each document and another one to slice them into chunks, get embeddings and set up a vector store for a RAG chat bot. At the moment it’s using Mixtral and the comman…

> layout detection with Amazon

Amazon Textract*.

It got lost in editing somehow

Re: Ollama now supports AMD graphics cards

#226

Post by Zambyte »

mejutoco wrote: post #224
Zambyte wrote: post #51

Earlier quoted context omitted.

Exactly. My friend was telling me that I was making a mistake for getting a 7900 XTX to run language models, when the fact of the matter is the cheapest NVIDIA card with 24 GB of VRAM is over 50% more expensive than the 7900 XTX. Running a high quality model at like 80 tps is way more important to me than running a way lower quality model at like 120 tps.

I cannot parse this. The Radeon RX 7900 XTX also has 24GB of vram, so how does it help you run higher quality models? I would understand if it had more ram.

Only the RX 7900 XTX has 24 GB of VRAM at its price point. If I went with an NVIDIA card, I would either have to spend over 50% more on the card, or use much worse models to fit on their 16 GB cards.

Re: Ollama now supports AMD graphics cards

#228

Post by UncleEntity »

cjbprime wrote: post #183
jameshart wrote: post #135

Earlier quoted context omitted.

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

This will likely build a version without GPU acceleration, I think?

I was trying to get AMD GPU support going in llama.cpp a couple weeks ago and just gave up after a while. 'rocminfo' shows that I have a GPU and, presumably, rocm installed but there were build problems I didn't feel like sorting out just to play with a LLM for a bit.

Kudos if Ollama has this sorted out.

Re: Ollama now supports AMD graphics cards

#229

Post by Retr0id »

Rising6378 wrote: post #204
throawayonthe wrote: post #107

Earlier quoted context omitted.

I’ve already been using ollama with my 6700xt just fine, you just have to set some env variable to make rocm work “unoficially” The linked page says they will support more soon, so i’m guessing this will just be integrated

which env variable did you set to what? currently I'm struggeling setting it up

HSA_OVERRIDE_GFX_VERSION=10.3.0 seems to be the magic words.

You can invoke it directly via `HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve`, but I added this line to the systemd unit at /etc/systemd/system/ollama.service:

    Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Re: Ollama now supports AMD graphics cards

#230

Post by haolez »

verdverm wrote: post #144
haolez wrote: post #121

Is there an equivalent to ollama or gpt4all for Android? I'd like to host my model somewhere and talk to it via app.

At that point, it sounds like your after an API endpoint for any model. Lots of solutions out there, depends more on your hosting

No, I want the opposite: I want a generic chatbot app that connects to any endpoint.
Post reply on HN