Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

llama.cpp

llama.app

Post reply on HN
Print view View original

171–177 of 177 posts

Re: llama.cpp

#171

Post by tredre3 »

cptskippy wrote: post #167
jwr wrote: post #97

Earlier quoted context omitted.

Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.

Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

Re: llama.cpp

#172

Post by LeBit »

greenmoonx wrote: post #156
broodbucket wrote: post #137

Earlier quoted context omitted.

llama.cpp is better and it's what everyone switches to after dipping their toes in with ollama, I don't get your point

If it's so good then why is Ollama way faster and 4x as popular as llama.cpp lol Beating you at your own game If Ollama is so bad then why did llama.cpp had to rip off their entire site, branding, even down to their install method

This is a really bizarre thread.

If you think Ollama is better, use it!

People who have used them all converge on llama.cpp. You don’t have to.

Re: llama.cpp

#173

Post by LeBit »

pinkmoonx wrote: post #121
mdp2021 wrote: post #46

Earlier quoted context omitted.

> Is llama.cpp (and thus llama.app) llama.app is just an URL (for the "advertisement" webpages of llama.cpp outside GitHub). > that much better than Ollama llama.cpp is the real thing, ollama was a fork that remained inferior.

Nah Ollama is way more friendly to use, glad they forked it. Love having options

Ollama is not competitive.

Re: llama.cpp

#174

Post by cptskippy »

tredre3 wrote: post #171
cptskippy wrote: post #167

Earlier quoted context omitted.

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me. Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.

Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.

I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.

Re: llama.cpp

#175

Post by mirekrusin »

LoganDark wrote: post #163
mirekrusin wrote: post #157

Earlier quoted context omitted.

Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins. ps. ngram didn't work for me very well, but dedicated speculative model works very well ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like: # over baseline at temp 0.6 (95…

I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.

Yeah for that dense models are not great better to pick some MoE to get benefits of large vram pool while keeping reasonable speed.

Re: llama.cpp

#176

Post by greenmoonx »

LeBit wrote: post #172
greenmoonx wrote: post #156

Earlier quoted context omitted.

If it's so good then why is Ollama way faster and 4x as popular as llama.cpp lol Beating you at your own game If Ollama is so bad then why did llama.cpp had to rip off their entire site, branding, even down to their install method

This is a really bizarre thread. If you think Ollama is better, use it! People who have used them all converge on llama.cpp. You don’t have to.

> If you think Ollama is better, use it!

It's not that simple. An army of dweebs from HN will harass you if they find out you're using Ollama. It's getting to the point you can't use it at work.

They all link the same article. I never care about it.

Forking software was one of the original ideas with GitHub - it's called Ollama. Anyone curious enough to deep dive can find out about the name, and how Ollama came to be.

Re: llama.cpp

#177

Post by macwhisperer »

pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...

I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak

Post reply on HN