[flagged]
llama.cpp
31–40 of 177 posts
Re: llama.cpp
#32llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…
So are there any alternatives which do actually work well with ROCm OOTB?
Re: llama.cpp
#33Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…
You're not wrong about the appearance of sketchiness of that, but.... dude, come on. "It's not hard" is only true because you already know how to do it.
If you’re unsure how to do it, you can ask others for help or Claude will tell you.
Point Claude Code at a repository and ask how to install it safely. You don’t have to know about make or cryptography of HTTPS or anything, really. It will walk you through the options and risk.
If you have questions about any part of it—i.e. you don’t recognize an acronym or deeply understand why something works—you can ask.
Or ask here! HN is filled with smart humans.
Re: llama.cpp
#34Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…
What would you win? Cmake is capable of executing shell commands and you end up with a binary that will execute on your machine anyway. At the end of the day it is just a matter of trust anyway, isn't it? I personally use bwrap if I'm not confident about the source. Cloning a repo and building it is not _that_ hard, but easy installation is often the thing that makes or breaks a product. I believe Ollama proves that…
Mostly agree but at least with git clone you have a hash and the malicious code has to be served to everyone, and GitHub is less likely to get hijacked by a malicious third party.
But yes, still trusting the project with arbitrary code execution on your machine, including build formulas that pull stuff from the internet and suffer from all the above anyways
Re: llama.cpp
#35Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
ollama uses llama.cpp
Re: llama.cpp
#36Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.
Re: llama.cpp
#37Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
does your comment depend on the OS? I thought MLX has better performance on MacOS than llama.cpp
Re: llama.cpp
#38Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
"Friends don't let friends use ollama" https://sleepingrobots.com/dreams/stop-using-ollama/
Re: llama.cpp
#39It seems that llama.app is a direct competitor to ollama.com I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
Ollama uses the llama.cpp backend for inference. I find Ollama noticably slower. Llama.cpp has had a built-in webui (used as llama-server) for a long time now so have owned the user experience too.
Re: llama.cpp
#40It seems that llama.app is a direct competitor to ollama.com I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
And ollama were sketchy about not providing proper credit to llama.cpp, even though that’s all they are, a wrapper for it.