111–120 of 179 posts
You can try it on Android right now: Download the Edge Gallery apk from github: https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 Download one of the .task files from huggingface: https://huggingface.co/collections/google/gemma-3n-preview-6... Import the .task file in Edge Gallery with the + bottom right. You can take pictures right from the app. The model is indeed pretty fast.
Earlier quoted context omitted.
I did the same thing on my Pixel Fold. Tried two different images with two different prompts: "What can you see?" and "Describe this image" First image ('Describe', photo of my desk) - 15.6 seconds to first token - 2.6 tokens/second - Total 180 seconds Second image ('What can you see?', photo of a bowl of pasta) - 10.3 seconds to first token - 3.1 tokens/second - Total 26 seconds The Edge Gallery app defaults to CPU…
Pixel 4a release date = August 2020 Pixel Fold was in the Pixel 8 generation but uses the Tensor G2 from the 7s. Pixel 7 release date = October 2022 That's a 26 month difference, yet a full order of magnitude difference in token generation rate on the CPU. Who said Moore's Law is dead? ;)
Earlier quoted context omitted.
LLMs neither understand nor reason, that has been shown multiple times.
Seems like some don’t like that LLMs aren’t really intelligent. https://neurosciencenews.com/llm-ai-logic-27987/
Ask these questions again in two years when the next winter happens.
Earlier quoted context omitted.
Thanks for this guide it's great. Okay perhaps my phones not great and perhaps this isn't optimized/pruned for phone use but it's unusably slow. The answers are solid from my brief test. I wouldn't exactly say phone use, unless you have no internet and you don't mind a bit of a wait. Really impressive, regardless.
What phone are you using?
You can try it on Android right now: Download the Edge Gallery apk from github: https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 Download one of the .task files from huggingface: https://huggingface.co/collections/google/gemma-3n-preview-6... Import the .task file in Edge Gallery with the + bottom right. You can take pictures right from the app. The model is indeed pretty fast.
I assume that "pretty fast" depends on the phone. My old Pixel 4a ran Gemma-3n-E2B-it-int4 without problems. Still, it took over 10 minutes to finish answering "What can you see?" when given an image from my recent photos. Final stats: 15.9 seconds to first token 16.4 tokens/second prefill speed 0.33 tokens/second decode speed 662 seconds to complete the answer
CPU:
7.37 seconds to first token
35.55 tokens/second prefill speed
7.09 tokens/second decode speed
27.97 seconds to complete the answer
GPU:
1.96 seconds to first token
133.40 tokens/second prefill speed
7.95 tokens/second decode speed
14.80 seconds to complete the answer
Earlier quoted context omitted.
I simplified what i wrote. There is an off accelerator memory where the embeddings are stored and queried at inference time, i did not want to get into details. That is how you reduce the in memory RAM. There are definitely more things going on in the paper as it builds upon the concept I described. The central idea remains the same: you have input embedding layers which map text to continuous vectors. Instead of loa…
The name "per-layer embeddings" is all we have to go on, and there are currently no published papers (that I'm aware of) using any similar mechanism, so, yes, it's a huge leap from a paper that doesn't mention per-layer anything. It's fine to speculate based on the name, but don't pretend that it's a known technique when it clearly isn't.
I have not looked at the model but since the embedding dimension of 256 seems quite small (for reference according to [3] the old Gemma 1B had 1152 dimension input embedding), I'm guessing that this is not done _in lieu_ of the main input embedding to first layer, but in addition to it.
[1] https://twitter.com/cccntu/status/1925043973170856393
[2] https://news.ycombinator.com/edit?id=44048662
[3] https://developers.googleblog.com/en/gemma-explained-whats-n...
Earlier quoted context omitted.
Imagine a model smarter than most humans that fits on your phone. edit: I seem to be the only one excited by the possibilities of such small yet powerful models. This is an iPhone moment: a computer that fits in your pocket, except this time it's smart.
I can't speak for anyone else, but these models only seem about as smart as google search, with enormous variability. I can't say I've ever had an interaction with a chatbot that's anything redolent of interaction with intelligence. Now would I take AI as a trivia partner? Absolutely. But that's not really the same as what I look for in "smart" humans.
Note that "smarter than smart humans" and "smarter than most humans" are not the same. The latter is a pretty low bar.
Earlier quoted context omitted.
It seems straightforward to me: Apps take up storage, and the only way to get more of that is to pay Apple's markups, as iOS devices don't support upgradable storage. On top of the already hefty markup, they don't even take storage capacity into consideration for trade-ins.
I am not aware of any phone allowing storage upgrades.