Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GGML – AI at the Edge

ggml.ai

Post reply on HN
Print view View original

111–120 of 246 posts

Re: GGML – AI at the Edge

#111

Post by SparkyMcUnicorn »

KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

Seems like serverless is the way to go for fast output while remaining inexpensive.

e.g.

https://replicate.com/stability-ai/stablelm-tuned-alpha-7b

https://github.com/runpod/serverless-workers/tree/main/worke...

https://modal.com/docs/guide/ex/falcon_gptq

Re: GGML – AI at the Edge

#112

Post by PrimeMcFly »

infamouscow wrote: post #100
PrimeMcFly wrote: post #24

Earlier quoted context omitted.

> I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship Do you have more info on the controversy? I'm not sure ejecting developers just because of controversy is honestly good stewardship.

The code is MIT licensed. If you don't agree with the direction the project is taking you can fork it and add whatever you want. I don't understand why this is so difficult for software developers with GitHub accounts to understand.

You've missed the point here more than I've seen anyone miss the point in a long time.

Re: GGML – AI at the Edge

#113

Post by sp332 »

nl wrote: post #56
moffkalast wrote: post #52

Earlier quoted context omitted.

Basically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.

According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.

It's the same memory footprint as inference. It's not that fast, and the paper mentions some optimizations that could still be done.

Re: GGML – AI at the Edge

#114

Post by binarymax »

Havoc wrote: post #67
binarymax wrote: post #54

Earlier quoted context omitted.

svantana is correct that PCs are edge, but if you meant "mobile", then ARM in iOS and Android typically have NEON instructions for SIMD, not AVX: https://developer.arm.com/Architectures/Neon

I was thinking more edge in the distributed serverless sense, but I guess for this type of use the compute part is slow not the latency so question doesn't make much sense in hindsight

Compute is the latency for LLMs :)

And in general, your inference code will be compiled to a CPU/Architecture target - so you can know ahead of time what instructions you'll have access to when writing your code for that target.

For example in the case of AWS Lambda, you can choose graviton2 (ARM with NEON), or x86_64 (AVX). The trick is that for some processors such as Xeon3+ there is AVX 512, and others you will top out at AVX 256. You might be able to figure out what exact instruction set your serverless target supports.

Re: GGML – AI at the Edge

#115

Post by halyconWays »

jart wrote: post #105
okhuman wrote: post #26

Earlier quoted context omitted.

+1. VC involvement in projects like these always pivot the team away from the core competency of what you'd expect them to deliver - into some commercialization aspect that convert only a tiny fraction of the community yet take up 60%+ of the core developer team's time. I don't know why project founders head this way...as the track records of leaders who do this end up disappointing the involved community at some poi…

Whenever a community project goes commercial, its interests are usually no longer aligned with the community. For example, llama.com makes frequent backwards-incompatible changes to its file format. I maintain a fork of ggml in the cosmopolitan monorepo which maintains support for old file formats. You can build and use it as follows: git clone https://github.com/jart/cosmopolitan cd cosmopolitan # cross-compile on x…

[dead]

Re: GGML – AI at the Edge

#116

Post by ljlolel »

yyyk wrote: post #68
ulchar wrote: post #46

Earlier quoted context omitted.

> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open sourc…

I've been going on and on about this in HN: Open source can win this fight, but I think OSS is overconfident. We need to be clear there are serious challenges ahead - ClosedAI and other corporations also have a plan, a plan that has good chances unless properly countered: A) Embed OpenAI (etc.) API everywhere. Make embedding easy and trivial. First to gain a small API/install moat (user/dev: 'why install OSS model wh…

Don’t forget chip shortages. That’s all centralized up through Nvidia, TSMC, and ASML

Re: GGML – AI at the Edge

#117

Post by b33j0r »

KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

I wish everyone in tech had your perspective. That is what I see, as well.

There is a lull right now between gpt4 and gpt5 (literally and metaphorically). Consumer models are plateauing around 40B for a barely-reasonable RTX 3090 (ggml made this possible).

Now is the time to launch your ideas, all!

Re: GGML – AI at the Edge

#118

Post by Miraste »

danielbln wrote: post #44
behnamoh wrote: post #39

Earlier quoted context omitted.

I wonder if ClosedAI and other companies use the findings of the open source community in their products. For example, do they use QLORA to reduce the costs of training and inference? Do they quantize their models to serve non-subscribing consumers?

Not disagreeing with your points, but saying "ClosedAI" is about as clever as writing M$ for Microsoft back in the day, which is to say not very.

M$ is a silly way to call Microsoft greedy. ClosedAI is somewhat better because OpenAI's very name is a bald-faced lie, and they should be called on it. Are there more elegant ways to do that? Sure, but every time I see Altman in the news crying crocodile tears about the "dangers" of open anything I think we need all the forms of opposition we can find.

Re: GGML – AI at the Edge

#120

Post by ignoramous »

rvz wrote: post #37
samwillis wrote: post #19

ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again.

A matter of when, not if. I mean, the website itself makes that much clear:

  The ggml way
  
    ...
  
    Open Core

    The library and related projects are freely available under the MIT license... In the future we may choose to develop extensions that are licensed for commercial use
  
    Explore and have fun!

    ... Contributors are encouraged to try crazy ideas, build wild demos, and push the edge of what's possible
So, like many other "open core" devtools out there, they'd like to have their cake and eat it too. And they might just as well, like others before them.

Won't blame anyone here though; because clearly, if you're as good as Georgi Gerganov, why do it for free?

Post reply on HN