Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
1–10 of 146 posts
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#2If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#3Not to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#4If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
I used gemm:2b with ollama last night and the model was around 1.3gb IIRC
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#5...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs.
Was not very impressed with the chat either.
So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#6If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
Depends how much you quantize the model. For most general-purpose LLMs, the model completely dwarfs the size of the binary code.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#7If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
https://ollama.com/library/gemma/tags
You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional.
The size of anything but the model is going to be rounding error compared to how large the models are, in this context.
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#8If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB.
In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#9Isn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI.
Has this perception changed or pretty much the same?
Re: Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models
#10...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
That does seem to be the consensus unfortunately. Would have been better for everyone if google’s foray into open model a la FB made a splash