Llama.cpp team has failed to make their tech easy to install and use for years.
Why can’t they figure it out???
llama.cpp
101–110 of 182 posts
Re: llama.cpp
#102Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
Re: llama.cpp
#103Earlier quoted context omitted.
> (Optimized by you through testing. Not that AI) Why not optimized by AI through testing ? Give it a test set to work on and let it loose.
AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.
^ This.
Intent is the answer and AI has none.
Re: llama.cpp
#104Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
ollama uses llama.cpp
The dev behind ollama is adamant that ollama doesn't use llama.cpp (based on a technicality -- it uses ggml, which is created by the same people behind llama.cpp and is the backend of llama.cpp)
He made such a big fuss about ollama implementing their own kernels and felt slighted about the online comments saying ollama didn't properly credit llama.cpp and it kind of left a bad taste in the mouth among the local inference community.
For me personally, it was this that made me avoid them at all costs: https://github.com/ollama/ollama/issues/11714#issuecomment-3...
Re: llama.cpp
#105Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
Re: llama.cpp
#106Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
Wow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.
I’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site
Re: llama.cpp
#107llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…
So are there any alternatives which do actually work well with ROCm OOTB?
Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.
Re: llama.cpp
#108Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…
[deleted]
Re: llama.cpp
#109Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…
You're not wrong about the appearance of sketchiness of that, but.... dude, come on. "It's not hard" is only true because you already know how to do it.
GP is saying "it's not hard", yet you seem to be hearing "you should already know how to do this without ever looking it up".
Those aren't the same assertions.
It's not a sensible assumption that a process must necessarily be difficult or complex just because you don't already know how to do it. There are an unenumerable number of tasks each of us don't know how to do and have never done before which are not difficult at all.
Re: llama.cpp
#110Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.