151–160 of 222 posts
Has anybody figured some of the best flags to compile llama.cpp for rocm? I'm using the framework desktop and the Vulkan backend, because it was easier to compile out of the box, but I feel there's large peformance gains on the table by swtiching to rocm. Not sure if installing with brew on ubuntu would be easier.
I think the biggest advantage for me with ollama is the ability to "hotswap" models with different utility instead of restarting the server with different models combined with the simple "ollama pull model". In other words, it has been quite convenient. Due to this post I had to search a bit and it seems that llama.cpp recently got router support[1], so I need to have a look at this. My main use for this is a discord…
Earlier quoted context omitted.
The second interpretation is nonsense of course. If you want GPL-like obligations, use the GPL. A license is what it says in the license, nothing extra. It's a legal document not a moral guideline. I do think it's a very good idea to always use the GPL (even though commercially minded types always get their panties in a bunch about the GPL) for any user-facing software, to force everybody to 'play fair and share'. Th…
MIT license lets you do what you want with the code. That's the deal. The blob storage thing is the real problem though. Nobody talks about it until they try to move their models somewhere else.
Earlier quoted context omitted.
I just hit that error a few minutes ago. I build my llama.cpp from source because I use CUDA on Linux. So I made the mistake of trying to run Gemma4 on an older version I had and I got the same error. It’s possible brew installs an older version which doens’t support Gemma4 yet.
And that's exactly why llama.cpp is not usable by casual users. They follow the "move fast and break things" model. With ollama, you just have to make sure you're getting/building the latest version.
For most users that wanted to run LLM locally, ollama solved the UX problem. One command, and you are running the models even with the rocm drivers without knowing. If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P
>solved the UX problem. >One command Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name`, and that running things in the terminal is already a gigantic UX blocker (Ollama's popularity comes from the fact that it has a GUI), why are you putting the blame back on an open source project that owes you approximately zero communication ?
There is a TON of difference. Ollama downloads the model from its own model library server, sticks it somewhere in your home folder with a hashed name and a proprietary configuration that doesn't use the in built metadata specified by the model creator. So you can't share it with any other tool, you can't change parameters like temp on the fly, and you are stuck with whatever quants they offer.
Earlier quoted context omitted.
Exactly. The blog post states that the alternatives listed are similarly intuitive. They are not. If you just need a chat app, then sure, there’s plenty of options. But if you want an OpenAI compatible API with model management, accessibility breaks down fast. I’m open to suggestions, but the alternatives outlined in the blog post ain’t it.
The reported alternatives seem pretty User-Friendly to me: > LM Studio gives you a GUI if that’s what you want. It uses llama.cpp under the hood, exposes all the knobs, and supports any GGUF model without lock-in. > Jan( https://www.jan.ai/ ) is another open-source desktop app with a clean chat interface and local-first design. > Msty( https://msty.ai/ ) offers a polished GUI with multi-model support and built-in RAG…
So no, they are not alternatives to ollama
The name "llama.cpp" doesn't seem very friendly anymore nowadays... Back then, "llama" probably referred to those models from Facebook, and now those Llama series models clearly can't represent the strongest open-source models anymore...
No mention of the fact that Ollama is about 1000x easier to use. Llama.cpp is a great project, but it's also one of the least user friendly pieces of software I've used. I don't think anyone in the project cares about normal users. I started with Ollama, and it was great. But I moved to llama.cpp to have more up-to-date fixes. I still use Ollama to pull and list my models because it's so easy. I then built my own set…
Earlier quoted context omitted.
Yeah my thoughts exactly. Definitely slop. I have no objection to using AI to help writing. I just don't want to read the same sloppy cliches again and again and again. The short sentences. The Bigger Picture. Here's the rub. It's not just A, it's B. It's like those cliche titles - for fun and profit, the unreasonable effectiveness of, all you need is, etc. etc. but throughout the prose. Stop it guys!
Can you share some excerpts from that article that feel LLM-written to you?
Not-this-but-that like "The local LLM ecosystem doesn’t need Ollama. It needs llama.cpp."
Weird signposting: "Benchmarks tell the story."
Heres-the-rub conclusion: "The Bigger Picture"
Starting every title with "The ...".
It's definitely largely human-written, but there are enough slop-isms to make it annoying to read. And of course it's totally possible for a human to write an an AI style, but that doesn't make it any less annoying.