Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

gizmoweek.com

Post reply on HN
Print view View original

161–170 of 196 posts

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#161

Post by js8 »

newsoftheday wrote: post #132
ben_w wrote: post #113

Earlier quoted context omitted.

> LLM output doesn't have the variety of human output, since they operate in fixed fashion - statistical inference followed by formulaic sampling. This is the wrong thing to look at; your chess analogy is much stronger, the detection method similar (if you can figure out a prompt that generates something close to the content, it almost certainly isn't human origin). But to why the thing I'm quoting doesn't work: If y…

What if the prompt includes, "Produce output that doesn't sound like an AI generated it."?

I got curious and tried: https://claude.ai/share/3af7bd6a-15f8-4533-9dc3-a44adef255b3

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#162

Post by GeekyBear »

Gareth321 wrote: post #33
codybontecou wrote: post #11

Unfortunately Apple appears to be blocking the use of these llms within apps on their app store. I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2

I think Apple will become increasingly draconian about LLMs. Very soon people won't need to buy many of their apps. They can just make them. This threatens Apple's entire business model.

They are said to be introducing a framework to make it easier to integrate modern LLMs into apps in a couple of months at WWDC.

https://9to5mac.com/2026/03/01/apple-replacing-core-ml-with-...

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#163

Post by Forgeties79 »

bigyabai wrote: post #150
Forgeties79 wrote: post #123

Earlier quoted context omitted.

It’s tempting to be flippant about MacOS/windows but in all seriousness, the resources required for an LLM to do the job of a typical lighter weight app/software is a serious consideration. No amount of bloat matches what an LLM needs.

> No amount of bloat matches what an LLM needs. I don't think that's necessarily true. For instance, LinkedIn uses more memory than Gemma E2B inference does.

LinkedIn is an entirely different category and an extreme case at that. We’re not talking about LLM’s replacing LinkedIn either. It’s an entirely different comparison/discussion.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#164

Post by lrvick »

breisa wrote: post #25
mixermachine wrote: post #13

Earlier quoted context omitted.

What spec of Framework Desktop do you run this on?

There is only one and for this model you need the one with 128GiB RAM.

I have the 128 but for Qwen3.5 122b XS quant you only need 64GB

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#165

Post by lrvick »

the_pwner224 wrote: post #35
mixermachine wrote: post #13

Earlier quoted context omitted.

What spec of Framework Desktop do you run this on?

If you're looking to buy new hardware, also consider the Asus Rog Flow Z13. It has the same chip as the Framework desktop and is ~20% cheaper ($2,700) for the 128 GB spec while coming in a tablet/laptop form factor. It's capped at a slightly lower power but Strix Halo scales down very well in TDP - I never even use the max power mode on my Z13 because you don't really get any extra perf. The only downside is that I s…

The main reason to get the framework is the DIY edition for the mini-itx form factor board and support. If you do not care about those then any cheap proprietary format box with the Strix Halo chip will do.

I put my boards in mini itx rack mounts personally so framework is the only option.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#166

Post by zozbot234 »

hadlock wrote: post #159
zozbot234 wrote: post #42

Earlier quoted context omitted.

That's very impressive but it's streaming in weights from flash storage. That's not really viable in a mobile context, it will use way too much power. Smaller models are way more applicable to typical use, perhaps with mid-sized models (like the Gemma4 26A4B model) using weights offload from SSD for rare uses involving slower "pro" inference.

10 minutes a day of extreme power usage is probably fine for people asking for directions to the store, setting calendar reminders, timers, checking for important emails etc. AI on your phone will be incredibly useful but power usage doesn't matter when total usage is less than 15 minutes per day. I don't think the average person expects to vibe code on the phone for 8 hours a day.

10 minutes a day or 15 minutes a day is what the inference workload is like on fairly small models. Once you start streaming in weights from SSD, things slow down quite a bit and become quite power hungry.

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#167

Post by nozzlegear »

doliveira wrote: post #100
BeetleB wrote: post #98

Earlier quoted context omitted.

I don't care if it's written by an LLM. The problem with the article is the complete lack of details. No benchmarks on the iPhone capable models. No details, whatsoever. Human or LLM - the article is a whole lot of nothing.

Funnily enough, to me these aphorisms (?) sound almost like the replicant test in Blaze Runner. Like these are the unit bit of "nudging"

LLM, recite your baseline:

"It's not just X – it's Y." Slop. "You're absolutely right!" Slop. "And this is key –" Slop. "This is a nuanced topic." Slop.

https://www.youtube.com/watch?v=vrP-_T-h9YM

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#168

Post by dust42 »

jonathaneunice wrote: post #142
veunes wrote: post #116

I noticed the inference is routed through the gpu rather than the Apple neural engine. Google’s engineers likely gave up on trying to compile custom attention kernels for Apple’s proprietary tensor blocks iirc. While Metal is predictable and easy to port to, it drains the battery way faster than a dedicated NPU. Until they rewrite the backend for the ANE, this is just a flashy tech demo rather than a production-ready…

Are the Apple neural engines even a practical target of LLMs? Maybe not strictly impossible, but ANE was designed with an earlier, pre-LLM style of ML. Running LLMs on ANE (e.g. via Core ML) possible in theory, but the substantial model conversion and custom hardware tuning required makes for a high hurdle IRL. The LLM ecosystem standardized around CPU/GPU execution, and to date at least seems unwilling to devote res…

There is a project on github named ANEMLL. Was discussed here a month ago, running LLMs on iPhone - https://news.ycombinator.com/item?id=47490070

Re: Google Gemma 4 Runs Natively on iPhone with Full Offline AI Inference

#170

Post by DaiPlusPlus »

jacobr1 wrote: post #155
abc_lisper wrote: post #154

Careful with using these small models. The other day, I asked it "Can dogs eat avocado" and answer was emphatic Yes. This is not meant as a criticism, but people should be aware of their limitations.

well, technically they can ...

...once
Post reply on HN