Ollama now supports AMD graphics cards • Hacker News Board

Re: Ollama now supports AMD graphics cards

#91

Post by jiggawatts » Fri, Mar 15, 2024, 8:47 PM UTC

jart wrote: post #55

programmarchy wrote: post #25

Earlier quoted context omitted.

Apple killed off OpenCL for their platforms when they created Metal which was disappointing. Sounds like ROCm will keep it alive but the fragmentation sucks. Gotta support CUDA, OpenCL, and Metal now to be cross-platform.

What is OpenCL? AMD GPUs support CUDA. It's called HIP. You just need a bunch of #define statements like this: #ifndef __HIP__ #include #include #else #include #include #define cudaSuccess hipSuccess #define cudaStream_t hipStream_t #define cudaGetLastError hipGetLastError #endif Then your CUDA code works on AMD.

Can you explain why nobody knows this trick, for some values of “nobody”?

Re: Ollama now supports AMD graphics cards

#92

Post by api » Fri, Mar 15, 2024, 8:50 PM UTC

airocker wrote: post #54

I heard "Nvidia for LLM today is similar to how Sun Microsystems was for the web"

... for a very brief period of time until Linux servers and other options caught up.

Re: Ollama now supports AMD graphics cards

#93

Post by api » Fri, Mar 15, 2024, 8:52 PM UTC

choilive wrote: post #15

deadalus wrote: post #7

I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.

They bump up VRAM because they can't compete on raw compute.

They lag on software a lot more than the lag on silicon.

Re: Ollama now supports AMD graphics cards

#94

Post by nerdix » Fri, Mar 15, 2024, 8:53 PM UTC

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

The answer to your question is:

ollama run mixtral

That's it. You're running a local LLM. I have no clue how to run llama.cpp

I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

Re: Ollama now supports AMD graphics cards

#95

Post by cbhl » Fri, Mar 15, 2024, 9:00 PM UTC

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

In my opinion, pre-built binaries and an easy-to-use front-end are things that should exist and are valid as a separate project unto themselves (see, e.g., HandBrake vs ffmpeg).

Using the name of the authors or the project you're building on can also read like an endorsement, which is not _necessarily_ desirable for the original authors (it can lead to ollama bugs being reported against llama.cpp instead of to the ollama devs and other forms of support request toil). Consider the third clause of BSD 3-Clause for an example used in other projects (although llama.cpp is licensed under MIT).

Re: Ollama now supports AMD graphics cards

#96

Post by superkuh » Fri, Mar 15, 2024, 9:05 PM UTC

KronisLV wrote: post #16

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

llama.cpp added first class support for the RX 580 by implementing the vulkan backend. There are some issues on older kernel amdgpu code where a llm process VRAM is never reloaded if it gets kicked out to GTT (in 5.x kernels) but overall it's much faster than the clBLAST opencl implementation.

Re: Ollama now supports AMD graphics cards

#97

Post by jart » Fri, Mar 15, 2024, 9:07 PM UTC

jiggawatts wrote: post #91

jart wrote: post #55

Earlier quoted context omitted.

What is OpenCL? AMD GPUs support CUDA. It's called HIP. You just need a bunch of #define statements like this: #ifndef __HIP__ #include #include #else #include #include #define cudaSuccess hipSuccess #define cudaStream_t hipStream_t #define cudaGetLastError hipGetLastError #endif Then your CUDA code works on AMD.

Can you explain why nobody knows this trick, for some values of “nobody”?

No idea. My best guess is their background is in graphics and games rather than machine learning. When CUDA is all you've ever known, you try just a little harder to find a way to keep using it elsewhere.

Re: Ollama now supports AMD graphics cards

#98

Post by mchiang » Fri, Mar 15, 2024, 9:10 PM UTC

Zambyte wrote: post #18

It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support. Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that the…

While the PRs went in slightly earlier, much of the time was spent on testing the integrations, and working with AMD directly to resolve issues.

There were issues that we resolved prior to cutting the release, and many reported by the community as well.

Re: Ollama now supports AMD graphics cards

#99

Post by jmorgan » Fri, Mar 15, 2024, 9:14 PM UTC

KronisLV wrote: post #16

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

The compatibility matrix is quite complex for both AMD and NVIDIA graphics cards, and completely agree: there is a lot of work to do, but the hope is to gracefully fall back to older cards.. they still speed up inference quite a bit when they do work!

Re: Ollama now supports AMD graphics cards

#100

Post by karmakaze » Fri, Mar 15, 2024, 9:21 PM UTC

aseipp wrote: post #70

karmakaze wrote: post #61

Earlier quoted context omitted.

It would serve Nvidia right if their insistence on only running CUDA workloads on their hardware results in adoption of ROCm/OpenCL.

You can use OpenCL just fine on Nvidia, but CUDA is just a superior compute programming model overall (both in features and design.) Pretty much every vendor offers something superior to OpenCL (HIP, OneAPI, etc), because it simply isn't very nice to use.

I suppose that's about right. The implementors are busy building on a path to profit and much less concerned about any sort-of lock-in or open standards--that comes much later in the cycle.