Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

llama.cpp

llama.app

Post reply on HN
Print view View original

91–100 of 184 posts

Re: llama.cpp

#91

Post by car »

This MacOS app used to be called LlamaBarn. Really excellent to see the fast progress being made.

Official repo, also has documentation how to configure server parameters:

https://github.com/ggml-org/Llama-macOS

Small tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.

Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.

Llama.cpp became part of Huggingface recently.

Re: llama.cpp

#92

Post by mrighele »

hypfer wrote: post #43

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…

> (Optimized by you through testing. Not that AI)

Why not optimized by AI through testing ? Give it a test set to work on and let it loose.

Re: llama.cpp

#93

Post by bjackman »

imrehg wrote: post #24

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost…

I think ROCm is just a total second class citizen in the space TBH.

It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here.

If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. giving OSS maintainers access to their board farms. Maybe they have already done that, in which case I guess I should say I'd double the size of that team.

Re: llama.cpp

#94

Post by reverius42 »

HelloUsername wrote: post #90
reverius42 wrote: post #25

Earlier quoted context omitted.

It's from https://github.com/ggml-org/llama.cpp -- not associated with Meta, it's been around for years, and surely they know about it -- so I would guess either it's not a trademark violation or they don't care.

> not associated with Meta, it's been around for year This post ( https://news.ycombinator.com/item?id=35100086 ) from march 2023 says in the title "Llama.cpp: Port of Facebook's LLaMA model in C/C++"

By "not associated with Meta" meant, as far as I know, the authors don't work at Meta -- not that llama.cpp is unrelated to Meta's llama model.

Re: llama.cpp

#95

Post by TekMol »

I tried

    curl -LsSf https://llama.app/install.sh | sh
and then

    llama serve -hf unsloth/Qwen3-4B-GGUF:Q4_0
Then I get:

    W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden
    Terminated
And the web interface says

    Server unavailable
Maybe it gets killed by the OS because it uses too much RAM?

When I try

    llama serve -hf unsloth/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
It seems to work. Nice.

Re: llama.cpp

#96

Post by LoganDark »

mrighele wrote: post #92
hypfer wrote: post #43

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…

> (Optimized by you through testing. Not that AI) Why not optimized by AI through testing ? Give it a test set to work on and let it loose.

AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.

Re: llama.cpp

#97

Post by jwr »

hypfer wrote: post #43

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…

Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.

Re: llama.cpp

#98

Post by redmoonx »

walrus01 wrote: post #9

Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…

[dead]

Re: llama.cpp

#99

Post by redmoonx »

hypfer wrote: post #43

Old news by now, but you might not be aware that llama-server can do multi-model for a while now, Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI) Then, any api client can just select a model and the system does th…

[dead]
Post reply on HN