41–50 of 146 posts
The velocity of the LLM open source ecosystem is absolutely insane. I just got into hobby projects with diffusion a week ago and I'm seeing non-stop releases. It's hard to keep up. It's a firehose of information, acronyms, code etc. It's been a great python refresher.
The velocity of the LLM open source ecosystem is absolutely insane. I just got into hobby projects with diffusion a week ago and I'm seeing non-stop releases. It's hard to keep up. It's a firehose of information, acronyms, code etc. It's been a great python refresher.
In fact it's probably better to dive deep into one hobby project like you're doing than constantly context switch with every little news item that comes up.
While working on gemma.cpp there were definitely a lot of "gee i wish i could clone myself and work on that other thing too".
Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
Not to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
Cool, any plans on adding K quants, an API server and/or a python wrapper? I really doubt most people want to use it as a cpp dependency and run models at FP16.
python wrapper - if you want to run the model in python I feel like there's already a lot of more mature options available (see the model variations at https://www.kaggle.com/models/google/gemma) , but if people really want this and have something they want to do with a python wrapper that can't be done with existing options let me know. (similar thoughts wrt to API servers).
Can any kind soul explain the difference between GGUF, GGML and all the other model packaging I am seeing these days? Was used to pth and the thing tf uses. Is this all to support inference or quantization? Who manages these formats or are they brewing organically?
My personal way of understanding it is this - the original sin of model weight format complexity is that NNs are both data and computation.
Representing the computation as data is the hard part and that's where the simplicity falls apart. Do you embed the compute graph? If so, what do you do about different frameworks supporting overlapping but distinct operations. Do you need the artifact to make training reproducible? Well that's an even more complex computation that you have to serialize as data. And so on..