Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama Turbo

ollama.com

Post reply on HN
Print view View original

231–240 of 251 posts

Re: Ollama Turbo

#231

Post by lxgr »

otabdeveloper4 wrote: post #186
api wrote: post #48

Earlier quoted context omitted.

> Repackaging existing software while literally adding no useful functionality was always their gig. Developers continue to be blind to usability and UI/UX. Ollama lets you just install it, just install models, and go. The only other thing really like that is LM-Studio. It's not surprising that the people behind it are Docker people. Yes you can do everything Docker does with Linux kernel and shell commands, but do y…

> Ollama lets you just install it, just install models, and go. So does the original llama.cpp. And you won't have to deal with mislabeled models and insane defaults out of the box.

Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.

Re: Ollama Turbo

#232

Post by wkat4242 »

vntok wrote: post #212
wkat4242 wrote: post #167

Earlier quoted context omitted.

But there's much less that works with that. OpenWebUI for example.

Open WebUI works perfectly fine with llama.cpp though. They have very detailed quick start docs on it: https://docs.openwebui.com/getting-started/quick-start/start...

Oh thanks I didn't know that :O

I do also need an API server though. The one built into OpenWebUI is no good because it always reloads the model if you use it first from the web console and then run an API call using the same model (like literally the same model from the workspace). Very weird but I avoid it for that reason.

Re: Ollama Turbo

#233

Post by guipsp »

am17an wrote: post #159
moralestapia wrote: post #5

Ollama is great but I feel like Georgi Gerganov deserves way more credit for llama.cpp. He (almost) single-handedly brought LLMs to the masses. With the latest news of some AI engineers' compensation reaching up to a billion dollars, feels a bit unfair that Georgi is not getting a much larger slice of the pie.

Seriously, people astroturfing this thread by saying ollama has a new engine. It literally is the same engine that llama.cpp uses and georgi and slaren maintain! VC funding will make people so dishonest and just plain grifters

No one is astroturfing. You cannot run any model with just GGML. It's a tensor library. Yes, it adds value, but I don't think that saying that ollama also does is unfair.

Re: Ollama Turbo

#234

Post by guipsp »

leopoldj wrote: post #225
mchiang wrote: post #35

Earlier quoted context omitted.

Thanks for the kind words. Since the new multimodal engine, Ollama has moved off of llama.cpp as a wrapper. We do continue to use the GGML library, and ask hardware partners to help optimize it. Ollama might look like a toy and what looks trivial to build. I can say, to keep its simplicity, we go through a deep amount of struggles to make it work with the experience we want. Simplicity is often overlooked, but we wan…

> Ollama has moved off of llama.cpp as a wrapper. We do continue to use the GGML library Where can I learn more about this? llama.cpp is an inference application built using the ggml library. Does this mean, Ollama now has it's own code for what llama.cpp does?

https://github.com/ollama/ollama/tree/main/model/models

Re: Ollama Turbo

#235

Post by otabdeveloper4 »

lxgr wrote: post #231
otabdeveloper4 wrote: post #186

Earlier quoted context omitted.

> Ollama lets you just install it, just install models, and go. So does the original llama.cpp. And you won't have to deal with mislabeled models and insane defaults out of the box.

Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.

Yes, of course it can.

Re: Ollama Turbo

#236

Post by buyucu »

zozbot234 wrote: post #207
buyucu wrote: post #193

More than one year in and Ollama still doesn't support Vulkan inference. Vulkan is essential for consumer hardware. Ollama is a failed project at this point: https://news.ycombinator.com/item?id=42886680

There's an open pull request https://github.com/ollama/ollama/pull/9650 but it needs to be forward ported/rebased to the current version before the maintainers can even consider merging it. Also realistically, Vulkan Compute support mostly helps iGPU's and older/lower-end dGPU's, which can only bring a modest performance speed up in the compute-bound preprocessing phase (because modern CPU inference wins in the text-…

That pull request has been open for more than a year. The owner rebased multiple times but eventually gave up because Ollama devs just don't care.

Re: Ollama Turbo

#237

Post by zozbot234 »

buyucu wrote: post #236
zozbot234 wrote: post #207

Earlier quoted context omitted.

There's an open pull request https://github.com/ollama/ollama/pull/9650 but it needs to be forward ported/rebased to the current version before the maintainers can even consider merging it. Also realistically, Vulkan Compute support mostly helps iGPU's and older/lower-end dGPU's, which can only bring a modest performance speed up in the compute-bound preprocessing phase (because modern CPU inference wins in the text-…

That pull request has been open for more than a year. The owner rebased multiple times but eventually gave up because Ollama devs just don't care.

That's not a helpful point of view. It's the contributors' job to keep a pull request up to date as the codebase evolves, a maintainer is under no obligation to accept a PR that has long become out of date and unmergeable.

Re: Ollama Turbo

#238

Post by romperstomper »

jnmandal wrote: post #29

I see a lot of hate for ollama doing this kind of thing but also they remain one of the easiest to use solutions for developing and testing against a model locally. Sure, llama.cpp is the real thing, ollama is a wrapper... I would never want to use something like ollama in a production setting. But if I want to quickly get someone less technical up to speed to develop an LLM-enabled system and run qwen or w/e locally…

It is weird but when I tried new gpt-oss:b20 model locally llama.cpp just failed instantly for me. At the same time under ollama it worked (very slow but anyway). I didn't find how to deal with llama.cpp but ollama definitely doing something under the hood to make models work.

Re: Ollama Turbo

#239

Post by rpdillon »

wkat4242 wrote: post #232
vntok wrote: post #212

Earlier quoted context omitted.

Open WebUI works perfectly fine with llama.cpp though. They have very detailed quick start docs on it: https://docs.openwebui.com/getting-started/quick-start/start...

Oh thanks I didn't know that :O I do also need an API server though. The one built into OpenWebUI is no good because it always reloads the model if you use it first from the web console and then run an API call using the same model (like literally the same model from the workspace). Very weird but I avoid it for that reason.

llama.cpp is what you want. It offers both a web UI and an API on the same port. I use llama.cpp's webui with gpt-oss-20b, and I also leverage it as an OpenAI-compatible server with gptel for Emacs. Very good product.

Re: Ollama Turbo

#240

Post by lxgr »

otabdeveloper4 wrote: post #235
lxgr wrote: post #231

Earlier quoted context omitted.

Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.

Yes, of course it can.

I wouldn't consider that a given at all, but apparently there's indeed `llama-server` which looks promising!

Then the only thing that's missing seems to be a canonical way for clients to instantiate that, ideally in some OS-native way (systemd, launchcd etc.), and a canonical port that they can connect to.

Post reply on HN