GGML – AI at the Edge • Hacker News Board

Re: GGML – AI at the Edge

#191

Post by orost » Wed, Jun 07, 2023, 3:48 AM UTC

Tan-Aki wrote: post #185

Can anyone explain to me, in simple terms, and at a high level, what the heck am I looking at? What is this library for? What does it mean "it is used by lama.cpp and whisper.cpp"? How is it revolutionary? Thank you very much in advance!

ggml is a library that provides operations for running machine learning models

llama.cpp is a project that uses ggml to run LLaMA, a large language model (like GPT) by Meta

whisper.cpp is a project that uses ggml to run Whisper, a speech recognition model by OpenAI

ggml's distinguishing feature is efficient operation on CPU. Traditionally, this sort of work is done on GPU, but GPUs with large amounts of memory are specialized and extremely expensive hardware. ggml achieves acceptable speed on commodity hardware.

Re: GGML – AI at the Edge

#192

Post by xiphias2 » Wed, Jun 07, 2023, 4:12 AM UTC

georgehotz wrote: post #148

xiphias2 wrote: post #101

Earlier quoted context omitted.

They are competing (although they are very different, tinygrad is full stack Python, ggml is focusing on a few very important models), but in my opinion George Hotz lost focus a bit by not working more on getting the low level optimizations perfect.

Which low level optimizations specifically are you referring to? I'm happy with most of the abstractions. We are pushing to assembly codegen. And if you meant things like matrix accelerators, that's my next priority. We are taking more a of breadth first approach. I think ggml is more depth first and application focused. (and I think Mojo is even more breadth first)

Maybe I'd love to see Tinygrad beat GGML in its own game (4 bit LLM support on M1 Mac GPU or Tensor cores) before adding more backends / models.

It's easy to debug because the generated kernels can be compared to GGML, and still gives something practical that we all can play with.

At this point breadth first is a bit boring, because this way we don't know how far tinygrad is from optimal generated output.

Re: GGML – AI at the Edge

#193

Post by noman-land » Wed, Jun 07, 2023, 4:14 AM UTC

Georgi if you're reading this, I've had a lot of fun with whisper.cpp llama.cpp because of you so thank you very much.

Re: GGML – AI at the Edge

#194

Post by legendofbrando » Wed, Jun 07, 2023, 4:25 AM UTC

Running whisper locally on my iPhone back in December and watching perfect transcriptions pop out without sending anything to a server was a real lightbulb moment for me that set in motion a bunch of the work I’m doing now. Excited to see the new heights this unlocks!

Re: GGML – AI at the Edge

#195

Post by kyt » Wed, Jun 07, 2023, 4:26 AM UTC

I used the GGML version of Whisper and I had to revert back to the PyTorch version released by OpenAI. The GGML version simply didn't work well even for the same model. I am assuming it has to do with the quantization.

Re: GGML – AI at the Edge

#196

Post by ankitg12 » Wed, Jun 07, 2023, 4:43 AM UTC

Quite impressive, able to run a LLM on my local mac

    % ./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "Let's talk about Machine Learning now"
    main: seed = 1686112244
    gpt2_model_load: loading model from 'models/gpt-2-117M/ggml-model.bin'
    gpt2_model_load: n_vocab = 50257
    gpt2_model_load: n_ctx   = 1024
    gpt2_model_load: n_embd  = 768
    gpt2_model_load: n_head  = 12
    gpt2_model_load: n_layer = 12
    gpt2_model_load: ftype   = 1
    gpt2_model_load: qntvr   = 0
    gpt2_model_load: ggml tensor size = 224 bytes
    gpt2_model_load: ggml ctx size = 384.77 MB
    gpt2_model_load: memory size =    72.00 MB, n_mem = 12288
    gpt2_model_load: model size  =   239.08 MB
    extract_tests_from_file : No test file found.
    test_gpt_tokenizer : 0 tests failed out of 0 tests.
    main: prompt: 'Let's talk about Machine Learning now'
    main: number of tokens in prompt = 7, first 8 tokens: 5756 338 1561 546 10850 18252 783

    Let's talk about Machine Learning now.

    The first step is to get a good understanding of what machine learning is. This is where things get messy. What do you think is the most difficult aspect of machine learning?

    Machine learning is the process of transforming data into an understanding of its contents and its operations. For example, in the following diagram, you can see that we use a machine learning approach to model an object.

    The object is a piece of a puzzle with many different components and some of the problems it solves will be difficult to solve for humans.

    What do you think of machine learning as?

    Machine learning is one of the most important, because it can help us understand how our data are structured. You can understand the structure of the data as the object is represented in its representation.

    What about data structures? How do you find out where a data structure or a structure is located in your data?

    In a lot of fields, you can think of structures as

    main: mem per token =  2008284 bytes
    main:     load time =   366.33 ms
    main:   sample time =    39.59 ms
    main:  predict time =  3448.31 ms / 16.74 ms per token
    main:    total time =  3894.15 ms

Re: GGML – AI at the Edge

#197

Post by KronisLV » Wed, Jun 07, 2023, 4:47 AM UTC

c_o_n_v_e_x wrote: post #150

KronisLV wrote: post #104

Just today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also…

What do you mean by commodity hardware? Single server single CPU socket x86/ARM boxes? Anything that does not have a GPU?

> What do you mean by commodity hardware?

In my case, my local workstation has a Ryzen 5 1600 desktop CPU from 2017 (first generation Zen, 14nm) and it still worked decently.

Of course, response times would grow with longer inputs and outputs or larger models, but getting a response in less than a minute when running off of purely CPU is encouraging in of itself.

Re: GGML – AI at the Edge

#198

Post by dangrover » Wed, Jun 07, 2023, 5:15 AM UTC

There's so much potential for this as a tech powering all sorts of products, I hope it doesn't just become some Tarsnap type thing. (judging from initial site)

Re: GGML – AI at the Edge

#199

Post by zo1 » Wed, Jun 07, 2023, 5:56 AM UTC

_20p0 wrote: post #170

zo1 wrote: post #158

Earlier quoted context omitted.

Really curious why you tried to rename the file format magic string to have your initials? Going from GGML (see Title of this post) to GGJT with JT being Justine Tunney? Seems quite unnecessary and bound to have rubbed a lot of people the wrong way. Here is the official commit undoing the change: https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...

Strange comment. This doesn't sound like a legitimate criticism because of "Here is the official commit undoing the change:" being not a link to a commit to start with, and secondly being a declined pull request, and thirdly for the reality that `master` writes `ggjt` header. Really looks like some axe-grinding here, if I'm being honest. Especially because it takes very little effort to find out what the present head…

I'm a dev and know how to use git. Honestly that's just the PR I could find whilst catching up on this story and was curious what this "magic string" was. The change is there and got reverted 100%, see the other commenter who was the one that made it. If there is a better link you're welcome to post it.

Re: GGML – AI at the Edge

#200

Post by ukuina » Wed, Jun 07, 2023, 6:12 AM UTC

ignoramous wrote: post #120

rvz wrote: post #37

Earlier quoted context omitted.

> ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. > I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve t…

> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. A matter of when , not if . I mean, the website itself makes that much clear: The ggml way ... Open Core The library and related projects are freely available under the MIT license... In the future we may choose to develop extensions that are licensed for commercial use Explore and have fun! ... Contr…

Sounds like the SQLite model, which has been a net positive for the computing world.