Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The local LLM ecosystem doesn’t need Ollama

sleepingrobots.com

Post reply on HN
Print view View original

181–190 of 222 posts

Re: The local LLM ecosystem doesn’t need Ollama

#181

Post by endymion-light »

brabel wrote: post #168
endymion-light wrote: post #164

Earlier quoted context omitted.

I'm making more of a pragmatic point. While ethics of companies are important, i'm still using OpenAI, Anthropic, Microsoft, Apple etc, so I definitely accept a trade-off between morality and ease-of-use. Currently i've found Ollama to have the best intuitive experience for trying new models. Once i've tried those models and decide on something to use for a project, I can deploy them, and not need to use a UI again.…

Keep in mind that as the post says, the model you’re trying via ollama may not be the model you asked for! And the performance may be subpar and not reflect the model true performance. Otherwise, I agree they offer an easy and polished product and that explains why they are so popular, besides their personal connections having resulted in their OpenAI partnership.

Good points - i'll definitely look into switching - just wanted to reflect on what's causing it to stay so popular.

Re: The local LLM ecosystem doesn’t need Ollama

#182

Post by alzoid »

Eisenstein wrote: post #156
well_ackshually wrote: post #69

Earlier quoted context omitted.

>solved the UX problem. >One command Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name`, and that running things in the terminal is already a gigantic UX blocker (Ollama's popularity comes from the fact that it has a GUI), why are you putting the blame back on an open source project that owes you approximately zero communication ?

> Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name` There is a TON of difference. Ollama downloads the model from its own model library server, sticks it somewhere in your home folder with a hashed name and a proprietary configuration that doesn't use the in built metadata specified by the model creator. So you can't share it with any other tool…

This was my issue with current client ecosystem. I get a .guff file. I should be able to open my AI Client of choice and File -> Open and select a .guff. Same as opening a .txt file. Alternatively, I have cloned a HF model, all AI Clients should automatically check for the HF cache folder.

The current offering have interfaces to HuggingFace or some model repo. They get you the model based on what they think your hardware can handle and save it to %user%/App Data/Local/%app name%/... (on windows). When I evaluated running locally I ended up with 3 different folders containing copies of the same model in different directory structures.

It seems like HuggingFace uses %user%/.cache/.. however, some of the apps still get the HF models and save them to their own directories.

Those features are 'fine' for a casual user who sticks with one program. It seems designed from the start to lock you into their wrapper. In the end they are all using llama cpp, comfy ui, openvino etc to abstract away the backed. Again this is fine but hiding the files from the user seems strange to me. If you're leaning on HF then why now use their own .cache?

In the end I get the latest llama.cpp releases for CUDA and SYCL and run llama-server. My best UX has been with LM Studio and AI Playground. I want to try Local AI and vLLM next. I just want control over the damn files.

Re: The local LLM ecosystem doesn’t need Ollama

#183

Post by zombot »

anakaine wrote: post #33
cientifico wrote: post #25

For most users that wanted to run LLM locally, ollama solved the UX problem. One command, and you are running the models even with the rocm drivers without knowing. If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P

Llama.cpp now has a gui installed by default. It previously lacked this. Times have changed.

I don't care about the GUI so much. Ollama lets me download, adjust and run a whole bunch of models and they are reasonably fast. Last time I compared it with Llama.cpp, finding out how to download and install models was a pain in Llama.cpp and it was also _much_ slower than Ollama.

Re: The local LLM ecosystem doesn’t need Ollama

#184

Post by shantnutiwari »

Just tried llama.cpp

NO, it is not simpler or even as simple as Ollama.

There are multiple options-- llama server and cli, its not obivous which model to use.

With ollama, its one file. And you get the models from their site, you can browse an easy list.

I dont have the time to go thru 20billlion hugging face models and decide which is the one for me.

Thanks, but I'm sticking with Ollama

Re: The local LLM ecosystem doesn’t need Ollama

#185

Post by alzoid »

ontouchstart wrote: post #165
tasuki wrote: post #134

Earlier quoted context omitted.

> This stuff is messy. Give them some credit. They give you an epic open source project. It seems to me the epic open source project was given to us by Georgi Gerganov. These people just tried to milk it for some money, and made everything a little worse in the process.

100%. UX is where the money is, it is in the wrapper, not the core. Unfortunately, the core is the most valuable and labor intensive part of it. With agentic coding, the gap between solid core and shitty wrapper is going to be wider and wider.

Especially when the solid core now ships with a web ui and API compatibility with OpenAI and Antropic. In my test of ai clients, Ollama was the only one I deleted.

Re: The local LLM ecosystem doesn’t need Ollama

#186

Post by ottah »

0xbadcafebee wrote: post #13

No mention of the fact that Ollama is about 1000x easier to use. Llama.cpp is a great project, but it's also one of the least user friendly pieces of software I've used. I don't think anyone in the project cares about normal users. I started with Ollama, and it was great. But I moved to llama.cpp to have more up-to-date fixes. I still use Ollama to pull and list my models because it's so easy. I then built my own set…

It's a lot better than it used to be and honestly way more powerful. If you haven't looked lately, you should https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...

Re: The local LLM ecosystem doesn’t need Ollama

#187

Post by Eisenstein »

alzoid wrote: post #182
Eisenstein wrote: post #156

Earlier quoted context omitted.

> Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name` There is a TON of difference. Ollama downloads the model from its own model library server, sticks it somewhere in your home folder with a hashed name and a proprietary configuration that doesn't use the in built metadata specified by the model creator. So you can't share it with any other tool…

This was my issue with current client ecosystem. I get a .guff file. I should be able to open my AI Client of choice and File -> Open and select a .guff. Same as opening a .txt file. Alternatively, I have cloned a HF model, all AI Clients should automatically check for the HF cache folder. The current offering have interfaces to HuggingFace or some model repo. They get you the model based on what they think your hard…

Check out Koboldcpp. The dev has a specific philosophy about things (minimal or no dependencies, no installers, no logs, don't do anything to user's system they didn't ask for explicitly) that I find particularly agreeable. It's a single exec and includes the kitchen sink so there is no excuse not to try it.

Re: The local LLM ecosystem doesn’t need Ollama

#188

Post by throwa356262 »

zombot wrote: post #183
anakaine wrote: post #33

Earlier quoted context omitted.

Llama.cpp now has a gui installed by default. It previously lacked this. Times have changed.

I don't care about the GUI so much. Ollama lets me download, adjust and run a whole bunch of models and they are reasonably fast. Last time I compared it with Llama.cpp, finding out how to download and install models was a pain in Llama.cpp and it was also _much_ slower than Ollama.

That is not true.

If you today visit a models page on huggingface, the site will show you the exact oneliner you need to run to it on llama.cpp.

I didn't measure it, but both download and inference felt faster than ollama. One thing that was definitely better was memory usage, which may be important if you want to run small models on SCB.

Re: The local LLM ecosystem doesn’t need Ollama

#189

Post by throwa356262 »

AndroTux wrote: post #179
homarp wrote: post #65

Earlier quoted context omitted.

like someone said above: brew install llama.cpp llama-server -hf ggml-org/gemma-4-E4B-it-GGUF --port 8000 (with MCP support and web chat interface) and you have OpenAI API on the same 8000 port. ( https://github.com/ggml-org/llama.cpp/tree/master/tools/serv... lists the endpoints)

And why do I use ggml-org/gemma-4-E4B-it-GGUF instead of one of the 162 other models that can be found under the ggml-org namespace? And how do I even know that this is the namespace to look at? That's what I meant by model management. I'm too tired to scroll through a bazillion models that all have very cryptic names and abbreviations just to find the one that works well on my system with my software stack. I want a…

1. Go to HF

2. Choose the model they recommend

3. Run the one-liner the site gives you

Bonus: faster access to latest models and better memory usage

Re: The local LLM ecosystem doesn’t need Ollama

#190

Post by yjftsjthsd-h »

mongrelion wrote: post #73
denismi wrote: post #54

Hmm.. pacman -Ss ollama | wc -l 16 pacman -Ss llama.cpp | wc -l 0 pacman -Ss lmstudio | wc -l 0 Maybe some day.

llama.cpp moves too quickly to be added as a stable package. Instead, you can get it directly from AUR: https://aur.archlinux.org/packages?O=0&K=llama.cpp There are packages for Vulkan, ROCm and CUDA. They all work.

That doesn't make sense. Why would llama.cpp need to move any faster than ollama? For that matter, why not have a llama.cpp package and llama.cpp-git in the AUR?
Post reply on HN