Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The local LLM ecosystem doesn’t need Ollama

sleepingrobots.com

Post reply on HN
Print view View original

101–110 of 222 posts

Re: The local LLM ecosystem doesn’t need Ollama

#101

Post by matja »

tosh wrote: post #84

This is a bit like saying stop using Ubuntu, use Debian instead. Both llama.cpp and ollama are great and focused on different things and yet complement each other (both can be true at the same time!) Ollama has great ux and also supports inference via mlx, which has better performance on apple silicon than llama.cpp I'm using llama.cpp, ollama, lm studio, mlx etc etc depending on what is most convenient for me at the…

> This is a bit like saying stop using Ubuntu, use Debian instead.

Not really, because Ubuntu has always acknowledged Debian and explicitly documented the dependency:

> Debian is the rock on which Ubuntu is built.

> Ubuntu builds on the Debian architecture and infrastructure and collaborates widely with Debian developers, but there are important differences. Ubuntu has a distinctive user interface, a separate developer community (though many developers participate in both projects) and a different release process.

Source: https://ubuntu.com/community/docs/governance/debian

Ollama never has for llama.cpp. That's all that's being asked for, a credit.

Re: The local LLM ecosystem doesn’t need Ollama

#102

Post by roosgit »

teekert wrote: post #63
nikodunk wrote: post #37

Earlier quoted context omitted.

Having read above article, I just gave llama.cpp a shot. It is as easy as the author says now, though definitely not documented quite as well. My quickstart: brew install llama.cpp llama-server -hf ggml-org/gemma-4-E4B-it-GGUF --port 8000 Go to localhost:8000 for the Web UI. On Linux it accelerates correctly on my AMD GPU, which Ollama failed to do, though of course everyone's mileage seems to vary on this.

Was hoping it was so easy :) But I probably need to look into it some more. llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4' llama_model_load_from_file_impl: failed to load model Edit: @below, I used `nix-shell -p llama-cpp` so not brew related. Could indeed be an older version indeed! I'll check.

I just hit that error a few minutes ago. I build my llama.cpp from source because I use CUDA on Linux. So I made the mistake of trying to run Gemma4 on an older version I had and I got the same error. It’s possible brew installs an older version which doens’t support Gemma4 yet.

Re: The local LLM ecosystem doesn’t need Ollama

#104

Post by ekianjo »

cientifico wrote: post #25

For most users that wanted to run LLM locally, ollama solved the UX problem. One command, and you are running the models even with the rocm drivers without knowing. If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P

> For most users that wanted to run LLM locally, ollama solved the UX problem

This does not absolve them from the license violation

Re: The local LLM ecosystem doesn’t need Ollama

#105

Post by zozbot234 »

roosgit wrote: post #102
teekert wrote: post #63

Earlier quoted context omitted.

Was hoping it was so easy :) But I probably need to look into it some more. llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4' llama_model_load_from_file_impl: failed to load model Edit: @below, I used `nix-shell -p llama-cpp` so not brew related. Could indeed be an older version indeed! I'll check.

I just hit that error a few minutes ago. I build my llama.cpp from source because I use CUDA on Linux. So I made the mistake of trying to run Gemma4 on an older version I had and I got the same error. It’s possible brew installs an older version which doens’t support Gemma4 yet.

And that's exactly why llama.cpp is not usable by casual users. They follow the "move fast and break things" model. With ollama, you just have to make sure you're getting/building the latest version.

Re: The local LLM ecosystem doesn’t need Ollama

#106

Post by endymion-light »

I'm sorry, on a mac, Ollama just works. It lets me use a model and test it quickly. This is like saying stop using google drive, upload everything to s3 instead!

When i'm using Ollama - I honeslty don't care about performance, I'm looking to try out a model and then if it seems good, place it onto a most dedicated stack specifically for it.

Re: The local LLM ecosystem doesn’t need Ollama

#108

Post by UqWBcuFx6NV4r »

FrozenSynapse wrote: post #45
cientifico wrote: post #25

For most users that wanted to run LLM locally, ollama solved the UX problem. One command, and you are running the models even with the rocm drivers without knowing. If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P

but if ollama is much slower, that's cutting on your fun and you'll be having better fun with a faster GUI

You’ve completely missed the point.

Re: The local LLM ecosystem doesn’t need Ollama

#109

Post by [deleted] »

tyfon wrote: post #6

I think the biggest advantage for me with ollama is the ability to "hotswap" models with different utility instead of restarting the server with different models combined with the simple "ollama pull model". In other words, it has been quite convenient. Due to this post I had to search a bit and it seems that llama.cpp recently got router support[1], so I need to have a look at this. My main use for this is a discord…

[deleted]
Post reply on HN