Come on Dejiko, we don't have time for this gema! https://www.youtube.com/watch?v=9FSAqDVZHhU
Every time I see Gemma all I hear is Jubei screaming Genmaaaa since the n is almost silent. https://youtu.be/TFR9-cZecWo?si=rMED2LEh-fssHeeG
Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
31–40 of 146 posts
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#32Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
Thanks for releasing this! What is your use case for this rather than llama.cpp? For the on-device AI stuff I mostly do, llama.cpp is better because of GPU/metal offloading.
llama.cpp is great, if it fit your needs you can use it. I think at this point llama.cpp is effectively a platform that's hardened for production.
In its current form, I think of gemma.cpp is more of a direct model implementation (somewhere between the minimalism of llama2.c and the generality of ggml).
I tend to think of 3 modes of usage:
- hacking on inference internals - there's very little indirection, no IRs, the model is just code, so if you want to add support for your own runtime support for sparsity/quantization/model compression/etc. and demo it working with gemma, there's minimal barriers to do so
- implementing experimental frontends - i'll add some examples of this in the very near future. but you're free to get pretty creative with terminal UIs, code that interact with model internals like the KV cache, accepting/rejecting tokens etc.
- interacting with the model locally with a small program - of course there's other options for this but hopefully this is one way to play with gemma w/ minimal fuss.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#33Awesome work on getting this done so quickly.
We just added Gemma to the HHEM leaderboard - https://huggingface.co/spaces/vectara/leaderboard, and as you can see there its doing pretty good in terms of low hallucination rate, relative to other small models.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#34...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#35If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
https://ollama.com/library/gemma/tags You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional. The size of anything but the model is going to be rounding error compared to how large the models a…
What's the use case of models this small? Can you use the "knowledge" encoded in them and ask them questions and get relevant answers, or are they used as text processors to summarize documents etc?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#36Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
> Although this is from Google, we're a very small team that wanted such a codebase to exist. We have lots of plans to use it ourselves and we hope other people like it and find it useful. This is really cool, Austin. Kudos to your team!
Thanks so much!
Everyone working on this self-selected into contributing, so I think of it less as my team than ... a team?
Specifically want to call out: Jan Wassenberg (author of https://github.com/google/highway) and I started gemma.cpp as a small project just a few months ago + Phil Culliton, Dan Zheng, and Paul Chang + of course the GDM Gemma team.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#37The velocity of the LLM open source ecosystem is absolutely insane.
I just got into hobby projects with diffusion a week ago and I'm seeing non-stop releases. It's hard to keep up. It's a firehose of information, acronyms, code etc.
It's been a great python refresher.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#38...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?
I eval'd it with vllm.
One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab.
Many Chinese llms have a similar "default sampling" issue.
But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#39Awesome work on getting this done so quickly. We just added Gemma to the HHEM leaderboard - https://huggingface.co/spaces/vectara/leaderboard , and as you can see there its doing pretty good in terms of low hallucination rate, relative to other small models.
> LLM hallucinations
I wasn't familiar with the term, good article - https://masterofcode.com/blog/hallucinations-in-llms-what-yo...
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#40Is this neutered in the way Gemini is (i.e. is the "censorship" built in) or is that a "feature" of the Gemini application?