111–120 of 182 posts
Earlier quoted context omitted.
You're not wrong about the appearance of sketchiness of that, but.... dude, come on. "It's not hard" is only true because you already know how to do it.
Exactly. That way of assuming everyone is a seasoned dev or sysadmin always baffles me. There are a ton of tinkerers out there, we are legion, and we are often reluctant to ask, especially here (as suggested by another parallel comment) because we don't want to receive an answer like this. Full circle.
Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…
I think ROCm is just a total second class citizen in the space TBH. It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here. If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. g…
Earlier quoted context omitted.
So are there any alternatives which do actually work well with ROCm OOTB?
Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing ca…
Also if I may ask, what does the rest of your stack look like (agent, harness etc)?
I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not
Llama.cpp team has failed to make their tech easy to install and use for years. Why can’t they figure it out???
Browse to served web page with chat UI....
By "easy" do you mean "very lazy"?
Llama.cpp team has failed to make their tech easy to install and use for years. Why can’t they figure it out???
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
"Friends don't let friends use ollama" https://sleepingrobots.com/dreams/stop-using-ollama/
Want people to use yours? Make it better