Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3n preview: Mobile-first AI

developers.googleblog.com

Post reply on HN
Print view View original

91–100 of 179 posts

Re: Gemma 3n preview: Mobile-first AI

#91

Post by the_pwner224 »

the_pwner224 wrote: post #90
philipkglass wrote: post #76

Earlier quoted context omitted.

I assume that "pretty fast" depends on the phone. My old Pixel 4a ran Gemma-3n-E2B-it-int4 without problems. Still, it took over 10 minutes to finish answering "What can you see?" when given an image from my recent photos. Final stats: 15.9 seconds to first token 16.4 tokens/second prefill speed 0.33 tokens/second decode speed 662 seconds to complete the answer

I did the same thing on my Pixel Fold. Tried two different images with two different prompts: "What can you see?" and "Describe this image" First image ('Describe', photo of my desk) - 15.6 seconds to first token - 2.6 tokens/second - Total 180 seconds Second image ('What can you see?', photo of a bowl of pasta) - 10.3 seconds to first token - 3.1 tokens/second - Total 26 seconds The Edge Gallery app defaults to CPU…

Pixel 4a release date = August 2020

Pixel Fold was in the Pixel 8 generation but uses the Tensor G2 from the 7s. Pixel 7 release date = October 2022

That's a 26 month difference, yet a full order of magnitude difference in token generation rate on the CPU. Who said Moore's Law is dead? ;)

Re: Gemma 3n preview: Mobile-first AI

#92

Post by angst »

tried out google/gemma-3n-E4B-it-litert-preview on galaxy s25 ultra

loads pretty fast. starts to reply near-instant (text chat mode).

doesn't answer questions like "when is your cutoff date"

apparently answers "may 15 2024" as today date so probably explains why it answered joe biden as answer to who is US president

Re: Gemma 3n preview: Mobile-first AI

#93

Post by happyopossum »

bilbo0s wrote: post #57
lxgr wrote: post #15

Earlier quoted context omitted.

Given Apple's track record in dealing with the problem of ballooning app sizes, I'm not holding my breath. The incentives are just not aligned – Apple earns $$$ on each GB of extra storage users have to buy.

I was thinking that the entire time I read HN User onlyrealcuzzo's comment. Why, on Earth, would Apple ever want to solve the problem of Apps taking up more space? That's just not good business. Way better business right now to put R&D into increased memory access speeds. Apple would need to have a different business model entirely for them to have a business case for fixing this. They may fix it because they just wa…

> Why, on Earth, would Apple ever want to solve the problem of Apps taking up more space?

To make their devices more pleasant / less frustrating to use.

They’ve got a long track record of introducing features that reduce app size, speed up installs, and reduce storage use from infrequently used apps - there’s no reason to believe they’d stop doing that except for cynical vitriol.

Re: Gemma 3n preview: Mobile-first AI

#94

Post by nsonha »

esafak wrote: post #5
onlyrealcuzzo wrote: post #2

Probably a better link: https://developers.googleblog.com/en/introducing-gemma-3n/ Gemma 3n is a model utilizing Per-Layer Embeddings to achieve an on-device memory footprint of a 2-4B parameter model. At the same time, it performs nearly as well as Claude 3.7 Sonnet in Chatbot Arena.

Imagine a model smarter than most humans that fits on your phone. edit: I seem to be the only one excited by the possibilities of such small yet powerful models. This is an iPhone moment: a computer that fits in your pocket, except this time it's smart.

You seem to be the only one expected that model to be "smarter than most human"

Leave that part out, I'm excited. I'd love to see this plays some roles in "inference caching", to reduce dependencies on external services.

If only agents can plan and match patterns of tasks locally, and only needs real intelligence for doing self-contained/computationally heavy tasks.

Re: Gemma 3n preview: Mobile-first AI

#95

Post by croes »

croes wrote: post #65
TeMPOraL wrote: post #30

Earlier quoted context omitted.

It's understanding.

LLMs neither understand nor reason, that has been shown multiple times.

Seems like some don’t like that LLMs aren’t really intelligent.

https://neurosciencenews.com/llm-ai-logic-27987/

Re: Gemma 3n preview: Mobile-first AI

#96

Post by abound »

lxgr wrote: post #31
barnas2 wrote: post #22

Is anyone able to test it via AiStudio? I pay for Google's AI subscription, but any attempt to use this model results in a message telling me I've hit my rate limit.

Same here. I've also seemingly hit a rate limit on Gemini Pro 2.5 (on an account not subscribed to Gemini Advanced) yesterday, even though my last query is weeks past. Possibly there's a capacity shortage (I'd presume it all runs on the same Google hardware in the end), and they are prioritizing paid inference?

I hit this yesterday even though my account is on Tier 2 or 3. In my case, the issue was that I was using an old model name (exp-03-25 or something) in requests. Update to the latest pro-preview-whatever and the rate limit issues should go away.

This sounds unintuitive, but in Google's defense the rate limit errors include a link to docs that explain this.

Re: Gemma 3n preview: Mobile-first AI

#97

Post by rhdjsjebshjffn »

hmapple wrote: post #60
rhdjsjebshjffn wrote: post #32

Earlier quoted context omitted.

I can't speak for anyone else, but these models only seem about as smart as google search, with enormous variability. I can't say I've ever had an interaction with a chatbot that's anything redolent of interaction with intelligence. Now would I take AI as a trivia partner? Absolutely. But that's not really the same as what I look for in "smart" humans.

Have you tried any SOTA models like o3? If not, I strongly encourage you to discuss your area of expertise with it and rate based on that It is incredibly competent

I'm not really sure what to look for, frankly. It makes a rather uninteresting conversation partner and its observations of the world bland and mealy-mouthed.

But potentially maybe I'm just not looking for a trivia partner in my software.

Re: Gemma 3n preview: Mobile-first AI

#99

Post by intelkishan »

nharada wrote: post #20
cmcconomy wrote: post #17

I'd love to see this deployable to edge that have a Google Coral TPU

Has Google continued releasing new versions of Coral? Seems like a new version with the latest TPU and enough memory specifically to support this model would be awesome for devs

Google has not released a new version since 2021. Even the SDK kit is not under active development(uses Python 3.8), since the last time I saw it.

Re: Gemma 3n preview: Mobile-first AI

#100

Post by int_19h »

ai-christianson wrote: post #3
onlyrealcuzzo wrote: post #2

Probably a better link: https://developers.googleblog.com/en/introducing-gemma-3n/ Gemma 3n is a model utilizing Per-Layer Embeddings to achieve an on-device memory footprint of a 2-4B parameter model. At the same time, it performs nearly as well as Claude 3.7 Sonnet in Chatbot Arena.

That seems way too good to be true. What's the catch?

The catch is that "does as good as X" is pretty much never representative of real world performance when it comes to LLMs.

In general, all those scores are mostly useful to filter out the models that are blatantly and obviously bad. But to determine whether the model is actually good at any specific thing that you need, you'll have to evaluate them yourself to find out.

Post reply on HN