11–20 of 146 posts
Isn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI. Has this perception changed or pretty much the same?
Not sure what you mean about Gemma considering it’s not a service. You can download the model weights and the inference code is on GitHub. Everything is local!
Isn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI. Has this perception changed or pretty much the same?
There are some very interesting efforts in JAX/TPU land like https://github.com/erfanzar/EasyDeL
To get a few common questions out of the way:
- This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an inspiration (we say as much in the README). I've been a big advocate of gguf + llama.cpp support for gemma and am happy for people to use that.
- how is it different than inference runtime X? gemma.cpp is a direct implementation of gemma, in its current form it's aimed at experimentation + research and portability + easy modifiable rather than a general purpose deployment framework.
- this initial implementation is cpu simd centric. we're exploring options for portable gpu support but the cool thing is it will build and run on a lot of environments you might not expect an llm to run, so long as you have the memory to load the model.
- I'll let other colleagues answer questions about the Gemma model itself, this is a C++ implementation of the model, but relatively independent of the model training process.
- Although this is from Google, we're a very small team that wanted such a codebase to exist. We have lots of plans to use it ourselves and we hope other people like it and find it useful.
- I wrote a twitter thread on this project here: https://twitter.com/austinvhuang/status/1760375890448429459
...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
That does seem to be the consensus unfortunately. Would have been better for everyone if google’s foray into open model a la FB made a splash
It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.
If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
Isn't there a huge risk that Google could most likely deprecate Gemini, Gemma and Gemma.cpp? Not really smart to build on anything with Google e.g. Google Cloud for AI. Has this perception changed or pretty much the same?
Come on Dejiko, we don't have time for this gema! https://www.youtube.com/watch?v=9FSAqDVZHhU
Earlier quoted context omitted.
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.
Pre LLM era (let's say 2020), the hardware used to look decently powerful for most use cases (disks in hundreds of GBs, dozen or two of RAM and quad or hex core processors) but with the advent of LLMs, even disk drives start to look pretty small let alone compute and memory.