does anyone have stats on cpu only inference speed with this?
any particular hardware folks are most interested in?
Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
51–60 of 146 posts
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#52Is it not possible to add Gemma support on Llama.cpp?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#53Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that. - how is it different than inference runtime X? ge…
Kudos on your release! I know this was just made available but
- Somewhere the README, consider adding the need for a `-DWEIGHT_TYPE=hwy::bfloat16_t` flag for non-sfp. Maybe around step 3.
- The README should explicitly say somehere that there's no GPU support (at the moment)
- "Failed to read cache gating_ein_0 (error 294)" is pretty obscure. I think even "(error at line number 294)" would be a big improvement when it fails to FindKey.
- There's something odd about the 2b vs 7b model. The 2b will claim its trained by Google but the 7b won't. Were these trained on the same data?
- Are the .sbs weights the same weights as the GGUF? I'm getting different answers compared to llama.cpp. Do you know of a good way to compare the two? Any way to make both deterministic? Or even dump probability distributions on the first (or any) token to compare?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#54Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#55Is it not possible to add Gemma support on Llama.cpp?
Gemma support has been added to llama.cpp, in fact it was added almost immediately after the release:
https://twitter.com/ggerganov/status/1760293079313973408
However, be aware that there were some quality issues with quantization initially (hopefully they're resolved but i haven't followed too closely): https://twitter.com/ggerganov/status/1760418864418934922
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#56Awesome work on getting this done so quickly. We just added Gemma to the HHEM leaderboard - https://huggingface.co/spaces/vectara/leaderboard , and as you can see there its doing pretty good in terms of low hallucination rate, relative to other small models.
> LLM hallucinations I wasn't familiar with the term, good article - https://masterofcode.com/blog/hallucinations-in-llms-what-yo...
Karpathy offers a more concise (and whimsical) explanation https://x.com/karpathy/status/1733299213503787018
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#57Not to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
I am confused how all these things are able to interoperate. Are the creators of these models following the same IO for their models? Won't the tokenizer or token embedder be different? I am genuinely confused by how the same code works for so many different models.
It's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don't work with the hyper optimized llama runtumes.
Also, there's a lot of magic going on behind the scenes with configs stored in gguf/huggingface format models, and the libraries that use them. There are different tokenizers, but they mostly follow the same standards.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#58Earlier quoted context omitted.
Cool, any plans on adding K quants, an API server and/or a python wrapper? I really doubt most people want to use it as a cpp dependency and run models at FP16.
There's a custom 8-bit quantization (SFP), it's what we recommend. At 16 bit, we do bfloat16 instead of fp16 thanks to https://github.com/google/highway , even on CPU. Other quants - stay tuned. python wrapper - if you want to run the model in python I feel like there's already a lot of more mature options available (see the model variations at https://www.kaggle.com/models/google/gemma ) , but if people really want…
In my experience there's really no reason to run any model above Q6_K, the performance is identical and you shave off almost 2 GB of VRAM of a 7B model compared to Q8. To those of us with single digit amounts, that's highly significant. But most people seem to go for 4 bits anyway and it's the AWQ standard too. If you think it'll make the model look bad, then don't worry, it's only the relative performance that matters.
I would think that having an OpenAI standard compatible API would be a higher priority over a python wrapper, since then it can act as a drop in replacement for most any backend.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#59Is this neutered in the way Gemini is (i.e. is the "censorship" built in) or is that a "feature" of the Gemini application?
[deleted]
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#60Earlier quoted context omitted.
There's a custom 8-bit quantization (SFP), it's what we recommend. At 16 bit, we do bfloat16 instead of fp16 thanks to https://github.com/google/highway , even on CPU. Other quants - stay tuned. python wrapper - if you want to run the model in python I feel like there's already a lot of more mature options available (see the model variations at https://www.kaggle.com/models/google/gemma ) , but if people really want…
In my experience there's really no reason to run any model above Q6_K, the performance is identical and you shave off almost 2 GB of VRAM of a 7B model compared to Q8. To those of us with single digit amounts, that's highly significant. But most people seem to go for 4 bits anyway and it's the AWQ standard too. If you think it'll make the model look bad, then don't worry, it's only the relative performance that matte…
A nice side effect of implementing cpu simd is you just need enough regular RAM, which tends to be far less scarce than VRAM. Nonetheless, I get your point that more aggressive quantization is valuable + will share with the modeling team.