Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

131–140 of 246 posts

Re: GGML – AI at the Edge

#131

Post by dr_dshiv »

SparkyMcUnicorn wrote: post #72
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

How does this work?

Re: GGML – AI at the Edge

#132

Post by regularfry »

sva_ wrote: post #110

Really impressive work and I've asked this before, but is it really a good thing to have basically the whole library in a single 16k line file?

It makes syncing between llama.cpp, whisper.cpp, and ggml itself quite straightforward.

I think the lesson here is that this setup has enabled some very high-speed project evolution or, at least, not got in its way. If that is surprising and you were expecting downsides, a) why; and b) where did they go?

Re: GGML – AI at the Edge

#133

Post by b33j0r »

Y_Y wrote: post #127
iamflimflam1 wrote: post #124

I've always thought on the edge to be IoT type stuff. So running on embedded devices. But maybe that not the case?

Like any new term the (mis)usage broadens the meaning over time until it either it's widely known, it's unfashionable, or most likely; it becomes so broad as to be meaningless and hence it achieves buzzword apotheosis. My old job title had "edge" in it, and I still don't know what it's supposed to mean, although "not cloud" is a good approximation.

Sounds like your job had a lot of velocity with lateral tragmorphicity in Q1, just in time for staff engineer optimization!

Nicely done. Here is ~$50 worth of stock.

Re: GGML – AI at the Edge

#134

Post by digitallyfree »

ignoramous wrote: post #77
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

OpenLLAMA will be released soon and it's 100% compatible with the original LLAMA.

https://github.com/openlm-research/open_llama

Re: GGML – AI at the Edge

#135

Post by camdenlock »

LoganDark wrote: post #90
csmpltn wrote: post #51

Earlier quoted context omitted.

> More details needed to know if this is good stewardship (ejecting two toxic individuals) or laziness (ejecting a villain and a hero to get rid of the "problem" easily). Man, nobody has time for this shit. Leave the games and the drama for the social justice warriors and the furries. People building shit ain't got time for this - ejecting trouble makers is the right way to go regardless of which "side" they're on.

> and the furries Um, what?

If you know, you know

Re: GGML – AI at the Edge

#137

Post by ignoramous »

chaxor wrote: post #130
ignoramous wrote: post #77

Earlier quoted context omitted.

Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.

Why is commercial necessary to run local models?

It isn't, but such models may eventually lag behind the FOSS ones.

Re: GGML – AI at the Edge

#138

Post by deet »

dr_dshiv wrote: post #131
SparkyMcUnicorn wrote: post #72

Earlier quoted context omitted.

Maybe I'm wrong, but I don't think you want it fine-tuned on your data. Pretty sure you might be looking for this: https://github.com/SamurAIGPT/privateGPT Fine-tuning is good for treating it how to act, but not great for reciting/recalling data.

How does this work?

The parent is saying that "fine tuning", which has a specific meaning related to actually retraining the model itself (or layers at its surface) on a specialized set of data, is not what the GP is actually looking for.

An alternative method is to index content in a database and then insert contextual hints into the LLM's prompt that give it extra information and detail with which to respond with an answer on-the-fly.

That database can use semantic similarity (ie via a vector database), keyword search, or other ranking methods to decide what context to inject into the prompt.

PrivateGPT is doing this method, reading files, extracting their content, splitting the documents into small-enough-to-fit-into-prompt bits, and then indexing into a database. Then, at query time, it inserts context into the LLM prompt

The repo uses LangChain as boilerplate but it's pretty easily to do manually or with other frameworks.

(PS if anyone wants this type of local LLM + document Q/A and agents, it's something I'm working on as supported product integrated into macOS, and using ggml; see profile)

Re: GGML – AI at the Edge

#139

Post by ignoramous »

ggerganov wrote: post #25
qeternity wrote: post #17

Earlier quoted context omitted.

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support

(a novice here who knows a couple of fancy terms)

> ...lately we have been augmenting it with GPU support.

Would you say you'd then be building an equivalent to Google's JAX?

Someone even asked if anyone would build a C++ to JAX transpiler [0]... I am wondering if that's something you may implement? Thanks.

[0] https://news.ycombinator.com/item?id=35475675

Re: GGML – AI at the Edge

#140

Post by s1k3s »

I'm out of the loop on this entire thing so call me an idiot if I get it wrong. Isn't this whole movement based on a model leak from Meta? Aren't licenses involved that prevent it from going commercial?
Post reply on HN