Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

llama.cpp

llama.app

Post reply on HN
Print view View original

41–50 of 178 posts

Re: llama.cpp

#41

Post by etdznots »

jakkos wrote: post #20
ur-whale wrote: post #12

Earlier quoted context omitted.

> Anything that suggests curl into bash just plain sketches me out. Yeah, 100% and it's becoming more and more of a thing, see rust install for example. OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on your Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)

> security concerns Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.

I think the best course is a docker container or a VM, i agree with the sentiment but trying to restrict the permissions of these things is fighting an uphill battle, it’s probably best to let them reign king in a disposable and isolated environment

Re: llama.cpp

#42

Post by imrehg »

d3Xt3r wrote: post #32
imrehg wrote: post #24

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…

So are there any alternatives which do actually work well with ROCm OOTB?

I just switched to Vulkan, and be done with it. :)

As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...

I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV

Re: llama.cpp

#43

Post by hypfer »

Old news by now, but you might not be aware that llama-server can do multi-model for a while now,

Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)

Then, any api client can just select a model and the system does the right thing.

It's great software. It just works.

__

You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.

Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.

Re: llama.cpp

#44

Post by [deleted] »

walrus01 wrote: post #9

Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…

[deleted]

Re: llama.cpp

#45

Post by etdznots »

imrehg wrote: post #24

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…

The first one multiple contributors highlighted the PR as urgent andits had lots of review but it appears to be waiting for another review and/or someone that owns the affected hardware to test that the PR fixes the issue, it wpuld be easy for you to test and report whether or not it does, and the second thing is not related to llama.cpp at all

Yes ideally there would be testing every hardware + software combo but this costs engineering time and $$$ money, and you are running on master branch, no master branch of any software is stable, inherently, if you run into issues, just stick to the old hash where stuff worked, why are you insistent on both being at the bleeding edge and experience 0 breakage!

Re: llama.cpp

#46

Post by mdp2021 »

prologic wrote: post #29

Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.

> Is llama.cpp (and thus llama.app)

llama.app is just an URL (for the "advertisement" webpages of llama.cpp outside GitHub).

> that much better than Ollama

llama.cpp is the real thing, ollama was a fork that remained inferior.

Re: llama.cpp

#47

Post by antonvs »

> No telemetry

Must be tough not to be able to monitor your own models!

(The odds that that tagline was AI-generated seem high.)

Re: llama.cpp

#48

Post by BloondAndDoom »

thebeardisred wrote: post #28
thecopy wrote: post #15

Earlier quoted context omitted.

>Anything that suggests curl into bash just plain sketches me out. How is it different than trusting any other method of installation? If URL has https and is from an author you trust i dont see the difference.

Package management provides cryptographic attestation over the entire process, including the scripts run. Nothing is arbitrary nor mutable in the default state. The files touched will be in a predictable place and the scope of privilege escalation is clearly defined.

But he’s not using a package, he’s downloading from gir repo and build which exactly same getting a bash script from the same repo and executing it.

Re: llama.cpp

#49

Post by epihelix »

walrus01 wrote: post #9

Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…

There are also prebuilt binary archives for just about any distribution and inference backend for the latest github release:

https://github.com/ggml-org/llama.cpp/releases

No need to compile unless you really need to.

Re: llama.cpp

#50

Post by quantumleaper »

itake wrote: post #37
karimf wrote: post #19

Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.

does your comment depend on the OS? I thought MLX has better performance on MacOS than llama.cpp

The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.
Post reply on HN