Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The local LLM ecosystem doesn’t need Ollama

sleepingrobots.com

Post reply on HN
Print view View original

191–200 of 222 posts

Re: The local LLM ecosystem doesn’t need Ollama

#191

Post by tmtvl »

denismi wrote: post #54

Hmm.. pacman -Ss ollama | wc -l 16 pacman -Ss llama.cpp | wc -l 0 pacman -Ss lmstudio | wc -l 0 Maybe some day.

Then again...

  zypper --no-refresh search llamacpp | tail -n5 | wc -l
  5
Sometimes Arch has the software you want at the version you want, other times it doesn't but other distros do. That's why there's half a billion distros instead of just one.

Re: The local LLM ecosystem doesn’t need Ollama

#192

Post by rf15 »

yjftsjthsd-h wrote: post #190
mongrelion wrote: post #73

Earlier quoted context omitted.

llama.cpp moves too quickly to be added as a stable package. Instead, you can get it directly from AUR: https://aur.archlinux.org/packages?O=0&K=llama.cpp There are packages for Vulkan, ROCm and CUDA. They all work.

That doesn't make sense. Why would llama.cpp need to move any faster than ollama? For that matter, why not have a llama.cpp package and llama.cpp-git in the AUR?

what are you talking about? llama.cpp doesn't need to respect ollamas speed at all. It does not depend on it, it's the opposite of that.

Re: The local LLM ecosystem doesn’t need Ollama

#193

Post by gettingoverit »

RobotToaster wrote: post #49
OtherShrezzing wrote: post #34

Earlier quoted context omitted.

While that might be true, for as long as its name is “.cpp”, people are going to think it’s a C++ library and avoid it.

It would make sense to just make the GUI a separate project, they could call it llama.gui.

It would make even more sense to rename it to ollama, get a copyright for the name, and see how thieves complain they've been robbed :>

Re: The local LLM ecosystem doesn’t need Ollama

#194

Post by cyanydeez »

Eisenstein wrote: post #180
cyanydeez wrote: post #161

Earlier quoted context omitted.

and Ollama suffered the same fate when wanting to try new models

What fate?

the impedance mismatch between when models are released and the capability of Ollama and other servers capability for use.

Re: The local LLM ecosystem doesn’t need Ollama

#195

Post by yjftsjthsd-h »

rf15 wrote: post #192
yjftsjthsd-h wrote: post #190

Earlier quoted context omitted.

That doesn't make sense. Why would llama.cpp need to move any faster than ollama? For that matter, why not have a llama.cpp package and llama.cpp-git in the AUR?

what are you talking about? llama.cpp doesn't need to respect ollamas speed at all. It does not depend on it, it's the opposite of that.

The claim was that llama.cpp moves too fast to be in Arch's normal repos. But Arch does package ollama. Therefore, either 1. ollama somehow avoids the need to move fast, or 2. it moves at an acceptable pace when packaged.

Edit: Or perhaps put differently: If ollama includes a copy of llama.cpp and has a non-AUR package, why can't there be a non-AUR package that's just llama.cpp without ollama?

Re: The local LLM ecosystem doesn’t need Ollama

#196

Post by bashbjorn »

Oh hey I'm also working on a thing to solve the devx of llama.cpp: https://github.com/nobodywho-ooo/nobodywho

In contrast to Ollama, this is a self-contained library, not a server.

I wrote some quick notes on this blogpost, just to jot down how we think about good open-source citizenship: https://www.nobodywho.ai/posts/notes-on-friends-dont-let-fri...

Re: The local LLM ecosystem doesn’t need Ollama

#197

Post by MyUltiDev »

The attribution and lock-in arguments are the loud parts of this story, but the quieter production reason to move is concurrency. llama.cpp's server takes parallel N with cont-batching enabled by default, which interleaves tokens from multiple requests inside a single batch and keeps the GPU busy. Ollama defaults its parallel slots low and the interaction is less transparent, so the first time three people share a single model instance you feel it before any of the ethics become relevant. For a 70B Q4_K_M on a workstation, the real ceiling is KV cache fragmentation, and you have to size the context window around the parallel count rather than around one user. What is the highest parallel value anyone here has kept stable on a 70B Q4_K_M before the cache eviction pattern starts hurting quality?

Re: The local LLM ecosystem doesn’t need Ollama

#198

Post by ryandrake »

alzoid wrote: post #182
Eisenstein wrote: post #156

Earlier quoted context omitted.

> Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name` There is a TON of difference. Ollama downloads the model from its own model library server, sticks it somewhere in your home folder with a hashed name and a proprietary configuration that doesn't use the in built metadata specified by the model creator. So you can't share it with any other tool…

This was my issue with current client ecosystem. I get a .guff file. I should be able to open my AI Client of choice and File -> Open and select a .guff. Same as opening a .txt file. Alternatively, I have cloned a HF model, all AI Clients should automatically check for the HF cache folder. The current offering have interfaces to HuggingFace or some model repo. They get you the model based on what they think your hard…

That's one of my major annoyances with the current state of local model infrastructure: All the cruft around what should be a simple matter of downloading and using a file. All these cache directories and file renaming and config files that point to all of these things. The special, bespoke downloading cli tools. It's just kind of awkward from the point of view of someone who is used to just using simple CLI tools that do one thing. Imagine if sqlite3 required all of these paths and hashes and downloaders and configs rather than letting you just run:

   sqlite3 my database.db

Re: The local LLM ecosystem doesn’t need Ollama

#199

Post by Eisenstein »

cyanydeez wrote: post #194
Eisenstein wrote: post #180

Earlier quoted context omitted.

What fate?

the impedance mismatch between when models are released and the capability of Ollama and other servers capability for use.

I'm a bit unsure what that has to do with someone running an outdated version of the program while trying to use a model that is supported in the latest release.

Re: The local LLM ecosystem doesn’t need Ollama

#200

Post by 4k93n2 »

shantnutiwari wrote: post #184

Just tried llama.cpp NO, it is not simpler or even as simple as Ollama. There are multiple options-- llama server and cli, its not obivous which model to use. With ollama, its one file. And you get the models from their site, you can browse an easy list. I dont have the time to go thru 20billlion hugging face models and decide which is the one for me. Thanks, but I'm sticking with Ollama

jan.ai would be the ideal route to take here then. its open source, has a simple chat interface, it uses llama.cpp, it lets you search for models and downloads them, and it supports .gguf so youre not locked in if you want to use the models with another program later on
Post reply on HN