Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

llama.cpp

llama.app

Post reply on HN
Print view View original

131–140 of 158 posts

Re: llama.cpp

#131

Post by elteto »

jakkos wrote: post #20
ur-whale wrote: post #12

Earlier quoted context omitted.

> Anything that suggests curl into bash just plain sketches me out. Yeah, 100% and it's becoming more and more of a thing, see rust install for example. OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on your Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)

> security concerns Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.

You are horrified it needs access to its own stuff? I mean, any software that supports plugins needs access to them, no? How is that unreasonable? And with bwrap that’s an extra cli option.

Re: llama.cpp

#132

Post by LoganDark »

mirekrusin wrote: post #119
LoganDark wrote: post #110

Earlier quoted context omitted.

Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.

Just run /goal to optimise it and you should be good in less than an hour. Also best to use models that support speculative decoding.

Optimize llama.cpp? Hmm.

WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.

Fable recommended n-gram speculation so I'm working on that now.

Re: llama.cpp

#133

Post by basedrum »

imrehg wrote: post #24

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…

I have a framework 13, but I couldn't imagine running a local llm on it, how do you do it? Do you have a eGPU?

Re: llama.cpp

#134

Post by dangoljames »

NorwegianDude wrote: post #112
timschumi wrote: post #105

Earlier quoted context omitted.

Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.

llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.

I use it on the daily. Haven't exactly missed ollama.

Re: llama.cpp

#135

Post by binary132 »

roenxi wrote: post #113
bjackman wrote: post #93

Earlier quoted context omitted.

I think ROCm is just a total second class citizen in the space TBH. It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here. If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. g…

You probably can just use Vulkan. Why don't you think Vulkan is an alternative? Can it not handle BLAS and memory access? What is a graphics card supposed to be doing that isn't efficient in Vulkan?

Sometimes it’s just that a particular backend is poorly optimized or has a regression on a particular platform as compared to the “mainstream” backends. For example, whisper.cpp’s Vulkan backend performs 2-3x worse than CPU on my Snapdragon X2 laptop when using the ggerganov v3 turbo model. It’s probably a simple fix, but it does need to be fixed.

Re: llama.cpp

#136

Post by walrus01 »

chaps wrote: post #129
walrus01 wrote: post #27

Earlier quoted context omitted.

It's literally three steps, assuming you have the equivalent of the debian "build-essential" dependencies installed on your system for cmake, compiler. It's the exact three rows of my post, pasted one at a time into the CLI. The llama-server build guide page is actually pretty good. I mean, sure, if there's people who can't figure that out, they're probably better off using a GUI that is a wrapper on top of somebody…

You're right about all of that up until the point any one of those commands doesn't work and you get a garglblargl amount of incomprehensible error messages. You've been in the sauce for so long that you've forgotten what it's like to start fresh in these things.

Please give it a try before you assume it won't work? Tell me what platform you're on and follow the three sequential cli commands I provided, and if it doesn't compile I'll be glad to look at your error message and get it going in the right direction.

Re: llama.cpp

#137

Post by broodbucket »

pinkmoonx wrote: post #120
HelloUsername wrote: post #38

Earlier quoted context omitted.

"Friends don't let friends use ollama" https://sleepingrobots.com/dreams/stop-using-ollama/

I wish you people would go away. Want people to use yours? Make it better

llama.cpp is better and it's what everyone switches to after dipping their toes in with ollama, I don't get your point

Re: llama.cpp

#138

Post by chaps »

walrus01 wrote: post #136
chaps wrote: post #129

Earlier quoted context omitted.

You're right about all of that up until the point any one of those commands doesn't work and you get a garglblargl amount of incomprehensible error messages. You've been in the sauce for so long that you've forgotten what it's like to start fresh in these things.

Please give it a try before you assume it won't work? Tell me what platform you're on and follow the three sequential cli commands I provided, and if it doesn't compile I'll be glad to look at your error message and get it going in the right direction.

You're missing the trees for the forest, friend. I know how to troubleshoot this stuff. I wouldn't expect some random schmo who's just learning ubuntu for the first time to be able to.

This, "just run these three commands" attitude is exactly the reason why these curl|sh "installers" have become popular.

Re: llama.cpp

#139

Post by dbgobrrr »

d3Xt3r wrote: post #32
imrehg wrote: post #24

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…

So are there any alternatives which do actually work well with ROCm OOTB?

Lemonade-server works pretty well (most of the time). It wraps llama.cpp and other runtimes - it downloads the official binaries as far as I could see, and you can set alternative versions if needed. Works nicely with Strix Halo for a while now.

https://lemonade-server.ai

Re: llama.cpp

#140

Post by Zetaphor »

emulio wrote: post #128
hypfer wrote: post #123

Earlier quoted context omitted.

If you revisit my comment and pay attention to the opener: > but you might not be aware that llama-server can do multi-model for a while now you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing. So the answer to the question has already been encoded as text available. _ I can see the desire for explicit validation though. For that, I…

btw, llama-swap provides a nice UI for monitoring performance and logs, and even the ability to stop an infinite session that consumes GPU resources (sometimes that happens). Does the llama.cpp UI provide the same? If not, it is too early to say that llama-swap is “superseded/no longer needed.”

It's also convenient if you want to run separate builds behind a single config, sometimes there's a PR I want to run before it's merged
Post reply on HN