Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

51–60 of 234 posts

Re: Ollama now supports AMD graphics cards

#51

Post by Zambyte »

coconut08 wrote: post #34
choilive wrote: post #15

Earlier quoted context omitted.

They bump up VRAM because they can't compete on raw compute.

it doesn't matter how much compute you have if you don't have enough vram to run the model.

Exactly. My friend was telling me that I was making a mistake for getting a 7900 XTX to run language models, when the fact of the matter is the cheapest NVIDIA card with 24 GB of VRAM is over 50% more expensive than the 7900 XTX. Running a high quality model at like 80 tps is way more important to me than running a way lower quality model at like 120 tps.

Re: Ollama now supports AMD graphics cards

#53

Post by londons_explore »

HarHarVeryFunny wrote: post #48
bravetraveler wrote: post #21

Earlier quoted context omitted.

I wouldn't read too much into support. It's more in terms of business/warranty/promises than what can actually do things I've had a 6900XT since launch and this is the first I'm hearing "unsupported" , having played with ROCM plenty over the years with Fedora Linux. I think, at most, it's taken a couple key environment variables

How hard would it be for AMD just to document the levels of support of different cards the way NVIDIA does with their "compute capability" numbers ?! I'm not sure what is worse from AMD - the ML software support they provide for their cards, or the utterly crap documentation. How about one page documenting AMD's software stack compared to NVIDIA, one page documenting what ML frameworks support AMD cards, and another…

And almost looks like they're deliberately trying to not win any market share.

It's as if the CEO is mates with NVidias CEO and has an unwritten agreement not to try too hard to topple the applecart...

Oh wait... They're cousins!

Re: Ollama now supports AMD graphics cards

#55

Post by jart »

programmarchy wrote: post #25
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

Apple killed off OpenCL for their platforms when they created Metal which was disappointing. Sounds like ROCm will keep it alive but the fragmentation sucks. Gotta support CUDA, OpenCL, and Metal now to be cross-platform.

What is OpenCL? AMD GPUs support CUDA. It's called HIP. You just need a bunch of #define statements like this:

    #ifndef __HIP__
    #include 
    #include 
    #else
    #include 
    #include 
    #define cudaSuccess hipSuccess
    #define cudaStream_t hipStream_t
    #define cudaGetLastError hipGetLastError
    #endif
Then your CUDA code works on AMD.

Re: Ollama now supports AMD graphics cards

#56

Post by zozbot234 »

moffkalast wrote: post #47
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

OpenCL is as dead as OpenGL and the inference implementations that exist are very unperformant. The only real options are CUDA, ROCm, Vulkan and CPU. And Vulkan is a proper pain too, takes forver to build compute shaders and has to do so for each model. It only makes sense on Intel Arc since there's nothing else there.

SYCL is a fairly direct successor to the OpenCL model and is not quite dead, Intel seems to be betting on it more than others.

Re: Ollama now supports AMD graphics cards

#57

Post by skipants »

harwoodr wrote: post #14
Apofis wrote: post #9

Earlier quoted context omitted.

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

ROCm: https://github.com/ollama/ollama/commit/6c5ccb11f993ccc88c47...

Another giveaway that it's ROCm is that it doesn't support the 5700 series...

I'm really salty because I "upgraded" to a 5700XT from a Nvidia GTX 1070 and can't do AI on the GPU anymore, purely because the software is unsupported.

But, as a dev, I suppose I should feel some empathy that there's probably some really difficult problem causing 5700XT to be unsupported by ROCm.

Re: Ollama now supports AMD graphics cards

#58

Post by mpreda »

moffkalast wrote: post #47
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

OpenCL is as dead as OpenGL and the inference implementations that exist are very unperformant. The only real options are CUDA, ROCm, Vulkan and CPU. And Vulkan is a proper pain too, takes forver to build compute shaders and has to do so for each model. It only makes sense on Intel Arc since there's nothing else there.

ROCm includes OpenCL. And it's a very performant OpenCL implementation.

Re: Ollama now supports AMD graphics cards

#59

Post by arein3 »

bornfreddy wrote: post #41
sevagh wrote: post #8

Earlier quoted context omitted.

Spoiler alert: not good enough to break CUDA's moat

Not sure why you're downvoted, but as far as I've heard AMD cards can't beat 4090 - yet. Still, I think AMD will catch or overtake NVidia in hardware soon, but software is a bigger problem. Hopefully the opensource strategy will pay off for them.

Really hope so, maybe this time will catch and last

Usually when corps open source stuff to get adoption, they stuff the adopters after they gain enough market share and the cycle repeats again

Re: Ollama now supports AMD graphics cards

#60

Post by duskwuff »

nottorp wrote: post #29
renewiltord wrote: post #24

Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.

Used crypto mining parts not available any more?

Crypto mining didn't require significant bandwidth to the card. Mining-oriented motherboards typically only provisioned a single lane of PCIe to each card, and often used anemic host CPUs (like Celeron embedded parts).
Post reply on HN