Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

gizmoweek.com

Post reply on HN
Print view View original

41–50 of 196 posts

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#41

Post by Forgeties79 »

Gareth321 wrote: post #33
codybontecou wrote: post #11

Unfortunately Apple appears to be blocking the use of these llms within apps on their app store. I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2

I think Apple will become increasingly draconian about LLMs. Very soon people won't need to buy many of their apps. They can just make them. This threatens Apple's entire business model.

I guess I am not seeing why would I want to abandon most (if any) of my simple, small, purpose-built apps that always do the exact thing I want for a private company’s ever-changing LLM that will approximate what I’m asking and approximate its response utilizing far more resources.

I’m sure there are things on my phone it could replace (though I struggle to think of them) but there are plenty it can’t. My black magic camera app, web browsers, local send, libby/hoopla…

I can’t really think of any apps I use every day - or every week - that an LLM would replace. I’m not coding on my smartphone and aside from that an LLM is basically a more complex, somewhat inconsistent search engine experience right now for most people. Siri didn’t replace any of my apps, for instance. Why would chatGPT?

TL;DR: what apps would an LLM replace on my iPhone?

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#42

Post by zozbot234 »

redbell wrote: post #29
karimf wrote: post #12

Related: Gemma 4 on iPhone (254 comments) - https://news.ycombinator.com/item?id=47652561

Another related submission from 22 days ago : iPhone 17 Pro Demonstrated Running a 400B LLM (+700pts, +300cmts): https://news.ycombinator.com/item?id=47490070

That's very impressive but it's streaming in weights from flash storage. That's not really viable in a mobile context, it will use way too much power. Smaller models are way more applicable to typical use, perhaps with mid-sized models (like the Gemma4 26A4B model) using weights offload from SSD for rare uses involving slower "pro" inference.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#43

Post by varispeed »

cyanydeez wrote: post #36
CubsFan1060 wrote: post #31

Earlier quoted context omitted.

Though of course Apple's rules aren't always consistent, I have 2 separate apps currently on my phone that can/are running this (Google's Edge Gallery and Locally AI)

Can't be just a SaaSpocolypse. LLMs with the right harness could obliterate much of the TODO+ apps with a general assistant. But it's more likely it's just walled garden + security theatre that'll keep them from allowing outside apps.

Wouldn't trust AI to run TODO, especially weak models. They can hallucinate tasks, forget to remind etc.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#44

Post by mrkpdl »

Gareth321 wrote: post #33
codybontecou wrote: post #11

Unfortunately Apple appears to be blocking the use of these llms within apps on their app store. I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2

I think Apple will become increasingly draconian about LLMs. Very soon people won't need to buy many of their apps. They can just make them. This threatens Apple's entire business model.

But… why would I put the effort into getting an llm to make me an app when a there’s an existing app that I don’t have to maintain? I don’t want to have to make every app I use?

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#49

Post by zozbot234 »

the_pwner224 wrote: post #38
bossyTeacher wrote: post #4

Is the output coherent though? I am yet to see a local model working on consumer grade hardware being actually useful.

I have a 128 GB Strix Halo tablet (same as the other commenter here with the Framework Desktop). I'm using the larger Gemma 4 26B-A4B model (only 28 GB @ Q8) and it's been working great and runs very fast. It's a 100% replacement for free ChatGPT/Gemini. Compared to the paid pro/thinking models... Gemma does have reasoning, and I have used the reasoning mode for some tax & legal/accounting advice recently as well as…

There's a 31B dense model in the Gemma 4 series that's obviously going to be smarter (though a whole lot slower) than the MoE 26A4B.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#50

Post by the_pwner224 »

zozbot234 wrote: post #49
the_pwner224 wrote: post #38

Earlier quoted context omitted.

I have a 128 GB Strix Halo tablet (same as the other commenter here with the Framework Desktop). I'm using the larger Gemma 4 26B-A4B model (only 28 GB @ Q8) and it's been working great and runs very fast. It's a 100% replacement for free ChatGPT/Gemini. Compared to the paid pro/thinking models... Gemma does have reasoning, and I have used the reasoning mode for some tax & legal/accounting advice recently as well as…

There's a 31B dense model in the Gemma 4 series that's obviously going to be smarter (though a whole lot slower) than the MoE 26A4B.

I tried it and it was unusably slow at ~5-6 TPS. 26A4B gets close to 40 TPS which is faster than you can read, and still pretty quick with reasoning enabled.
Post reply on HN