Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

231–235 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#231

Post by ecurtin »

discardedrefuse wrote: post #223
jdright wrote: post #77

Earlier quoted context omitted.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

I gave Ramalama shot today. I'm very impressed. `uvx ramalama run deepseek-r1:1.5b` just works™ for me. And that's saying A LOT, because I'm running Fedora Kinoite (KDE spin of Silverblue) with nothing layered on the ostree. That means no ROCm or extra AMDGPU stuff on the base layer. Prior to this, I was running llamafile in a podman/toolbox container with ROCm installed inside. Looks like the container ramalama is u…

One of the main goals of RamaLama at the start was to be easy to install and run for Silverblue and Kinoite users (and funnily enough that machine had an AMD GPU, so we had almost identical setups). I quickly realized contributing to Ollama wasn't possible without being an Ollama employee:

https://github.com/ollama/ollama/pulls/ericcurtin

They merged a one-line change of mine, but you can't get any significant PRs in.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#232

Post by discardedrefuse »

ecurtin wrote: post #231
discardedrefuse wrote: post #223

Earlier quoted context omitted.

I gave Ramalama shot today. I'm very impressed. `uvx ramalama run deepseek-r1:1.5b` just works™ for me. And that's saying A LOT, because I'm running Fedora Kinoite (KDE spin of Silverblue) with nothing layered on the ostree. That means no ROCm or extra AMDGPU stuff on the base layer. Prior to this, I was running llamafile in a podman/toolbox container with ROCm installed inside. Looks like the container ramalama is u…

One of the main goals of RamaLama at the start was to be easy to install and run for Silverblue and Kinoite users (and funnily enough that machine had an AMD GPU, so we had almost identical setups). I quickly realized contributing to Ollama wasn't possible without being an Ollama employee: https://github.com/ollama/ollama/pulls/ericcurtin They merged a one-line change of mine, but you can't get any significant PRs in…

I just realized that ramalama is actually part of the whole Container Tools ecosystem (Podman, Buildah, etc). This is excellent! Thanks for doing this.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#233

Post by buyucu »

bdhcuidbebe wrote: post #215
buyucu wrote: post #212

Earlier quoted context omitted.

I'm willing to bet that Vulkan will outperform OpenVINO. Vulkan is the API right now in the graphics world. It's very well supported and actively being improved on. Everyone is pouring resources into making Vulkan better. OpenVINO feels barely developed. Intel never made it a proper backend for Pytorch like AMD did with ROCm. It's hard to see where it is going, or if it is going anywhere at all. Between Sycl and OneA…

> I'm willing to bet that Vulkan will outperform OpenVINO. > Vulkan is the API right now in the graphics world. YUP, Vulkan is all the rage in the graphics world, and for good reasons. But we arent discussing graphics now are we? Vulkan is a general graphics API with some computing capabilities. OpenVINO is a toolkit for inference neural networks, by intel built to make use of their GPUs and NPUs for this specific ta…

OpenVINO is quasi-abandoned, nobody uses it, Intel barely puts any effort to improve it.

Vulkan is used by millions and huge money goes into optimizing it.

My money is on Vulkan.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#235

Post by jdright »

discardedrefuse wrote: post #223
jdright wrote: post #77

Earlier quoted context omitted.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

I gave Ramalama shot today. I'm very impressed. `uvx ramalama run deepseek-r1:1.5b` just works™ for me. And that's saying A LOT, because I'm running Fedora Kinoite (KDE spin of Silverblue) with nothing layered on the ostree. That means no ROCm or extra AMDGPU stuff on the base layer. Prior to this, I was running llamafile in a podman/toolbox container with ROCm installed inside. Looks like the container ramalama is u…

I'll try it then, if it can get a docker setup using my GPU and no dependency hell, then good. I'll report back to correct myself once I try it.
Post reply on HN