Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

llama.cpp

llama.app

Post reply on HN
Print view View original

171–180 of 193 posts

Re: llama.cpp

#171

Post by tredre3 »

cptskippy wrote: post #167
jwr wrote: post #97

Earlier quoted context omitted.

Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.

Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

Re: llama.cpp

#172

Post by LeBit »

greenmoonx wrote: post #156
broodbucket wrote: post #137

Earlier quoted context omitted.

llama.cpp is better and it's what everyone switches to after dipping their toes in with ollama, I don't get your point

[flagged]

This is a really bizarre thread.

If you think Ollama is better, use it!

People who have used them all converge on llama.cpp. You don’t have to.

Re: llama.cpp

#173

Post by LeBit »

pinkmoonx wrote: post #121
mdp2021 wrote: post #46

Earlier quoted context omitted.

> Is llama.cpp (and thus llama.app) llama.app is just an URL (for the "advertisement" webpages of llama.cpp outside GitHub). > that much better than Ollama llama.cpp is the real thing, ollama was a fork that remained inferior.

[dead]

Ollama is not competitive.

Re: llama.cpp

#174

Post by cptskippy »

tredre3 wrote: post #171
cptskippy wrote: post #167

Earlier quoted context omitted.

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way. https://github.com/mostlygeek/llama-swap

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me. Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.

Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.

I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.

Re: llama.cpp

#175

Post by mirekrusin »

LoganDark wrote: post #163
mirekrusin wrote: post #157

Earlier quoted context omitted.

Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins. ps. ngram didn't work for me very well, but dedicated speculative model works very well ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like: # over baseline at temp 0.6 (95…

I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.

Yeah for that dense models are not great better to pick some MoE to get benefits of large vram pool while keeping reasonable speed.

Re: llama.cpp

#177

Post by macwhisperer »

pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...

I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak

Re: llama.cpp

#179

Post by walrus01 »

pplonski86 wrote: post #36

Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model…

Just FYI lm-studio is a GUI wrapper on top of a copy of llama-server that the lm-studio developers compile and distribute

Re: llama.cpp

#180

Post by teleforce »

Check this DLLM D project, a minimal, clean coding agent built directly on llama.cpp without Python, bindings or overhead [1].

The blog on the DLLM development [2].

[1] DLLM:

https://github.com/DannyArends/DLLM

[2] Teaching an AI to Know Itself: Building a Local LLM Agent in D:

https://blog.dlang.org/2026/06/07/teaching-an-ai-to-know-its...

Post reply on HN