Is this model & architecture compatible with llama.cpp and friends?
Gemma 3n preview: Mobile-first AI
21–30 of 179 posts
Re: Gemma 3n preview: Mobile-first AI
#22Is anyone able to test it via AiStudio? I pay for Google's AI subscription, but any attempt to use this model results in a message telling me I've hit my rate limit.
Re: Gemma 3n preview: Mobile-first AI
#23I'd love to see this deployable to edge that have a Google Coral TPU
Has Google continued releasing new versions of Coral? Seems like a new version with the latest TPU and enough memory specifically to support this model would be awesome for devs
I looked into this recently. Looks like it's a "no".
However there are now alternatives like the official RPi AI Hat that has between about 3x to 6x the TOPs (4 for Coral Vs 13/26 for RPi depending on model) so there is that. 20 TOPs on a RPi 5 - complete with nice vertically integrated camera etc - is quite interesting.
Re: Gemma 3n preview: Mobile-first AI
#24According to the readme here - https://huggingface.co/google/gemma-3n-E4B-it-litert-preview
E4B has a score of 44.4 in the Aider polyglot dashboard. Which means its on-par with gemini-2.5-flash (not the latest preview but the version used for the bench on aider's website), gpt4o and gpt4.5.
Thats sounds very good - imagine what a coding focused version of this could do if this is a "generic" embedded only model.
On the other hand - this does have a much lower score for livecodebench.
Re: Gemma 3n preview: Mobile-first AI
#25Earlier quoted context omitted.
Thank you, that helped a bit, although it's still not clear what exactly those parameters _are_. "Per-Layer Embedding (PLE) parameters that are used during model execution to create data that enhances the performance of each model layer." is too vague, and I can't find any other reference to "per-layer embedding parameters" in literature.
A layer is a transformer block / layer (basically the building block of the modern LLM architectures) - maybe Gemini can help you: https://gemini.google.com/share/cc58a7c6089e
I am perfectly aware of that. I don't believe other LLMs have such embeddings per layer, only the usual weights, so these per-layer embeddings seem to be distinguished from weights in some way. Afaik trying to play the same "cache in fast storage and load on demand" wouldn't work with layer weights since you'd end up with too much back/forth (you'd touch every cached byte on each token, assuming no MoE), so I'm guessing these embeddings are structured in a way that's broken up by concept.
Re: Gemma 3n preview: Mobile-first AI
#26Earlier quoted context omitted.
https://ai.google.dev/gemma/docs/gemma-3n#parameters > Gemma 3n models are listed with parameter counts, such as E2B and E4B, that are lower than the total number of parameters contained in the models. The E prefix indicates these models can operate with a reduced set of Effective parameters. This reduced parameter operation can be achieved using the flexible parameter technology built into Gemma 3n models to help th…
Thank you, that helped a bit, although it's still not clear what exactly those parameters _are_. "Per-Layer Embedding (PLE) parameters that are used during model execution to create data that enhances the performance of each model layer." is too vague, and I can't find any other reference to "per-layer embedding parameters" in literature.
Thanks. It is a bit vague to me too. If you need to load 5B per token generation any way, what's that different from selective offloading technique where some MLP weights offloaded to fast storage and loaded during each token generation?
Re: Gemma 3n preview: Mobile-first AI
#27What is "Per Layer Embeddings"? The only hit I can find for that term is the announcement blogpost. And for that matter, what is >mix’n’match capability in Gemma 3n to dynamically create submodels It seems like mixture-of-experts taken to the extreme, where you actually create an entire submodel instead of routing per token?
Per layer LoRA adapters, perhaps? - same as Apple is using for on-device AI.
Re: Gemma 3n preview: Mobile-first AI
#28Re: Gemma 3n preview: Mobile-first AI
#29According to the readme here - https://huggingface.co/google/gemma-3n-E4B-it-litert-preview E4B has a score of 44.4 in the Aider polyglot dashboard. Which means its on-par with gemini-2.5-flash (not the latest preview but the version used for the bench on aider's website), gpt4o and gpt4.5. Thats sounds very good - imagine what a coding focused version of this could do if this is a "generic" embedded only model. On t…
[flagged]