121–130 of 196 posts
Unfortunately Apple appears to be blocking the use of these llms within apps on their app store. I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2
What is your app doing? Just LLM inference?
Basically, a "toy" app to showcase where we are with coding agents on-device.
Is it me, or does the article sound like LLM output? The pattern "It's not mere X — it's Y", occurs like 4 times in the text :v
It is like the AI is training us to avoid certain language patterns. I rebel at the hostage of weak language: for strong language is next.
Earlier quoted context omitted.
And the end results is.....? What? A todo app that takes 16GB of RAM?
Nothing that Mac and Windows users aren't already used to.
I really hope this is a preview of the replacement for Siri that Google is creating bc these models are fantastic for their size!
Apple is getting a base Gemini model (not a Gemma), and it will run on Apple private compute. Apple foundational models will remain the on device model
is there a comparison of it running on iPhone vs. Android phones?
Running Gemma-4-E2B-it on an iPhone 15 (can't go higher than that due to RAM limitations) versus a Pixel 9 Pro, I don't really notice much of a difference between the two. The Pixel is a bit faster, but also a year more recent. The model itself works absolutely fine, though the iPhone thermal throttles at some point which really reduces the token generation speed. When I asked it to write me a business plan for a fis…
I noticed the inference is routed through the gpu rather than the Apple neural engine. Google’s engineers likely gave up on trying to compile custom attention kernels for Apple’s proprietary tensor blocks iirc. While Metal is predictable and easy to port to, it drains the battery way faster than a dedicated NPU. Until they rewrite the backend for the ANE, this is just a flashy tech demo rather than a production-ready…
Running background processes might motivate the use of NPU more but don't exactly feel like a pressing need. Actively listen to you 24/7 and analyze the data isn't a usecase I'm eager to explore given the lack of control we have of our own devices.
Offline or not, I'm sure Google uploads every keystroke, phone orientation, photo, WiFi endpoints and your shoe size when you interact with it. To enhance your experience.
At a glance, I see they do gather analytics about how much the app is used (model downloads, model invocations etc) without message content, pretty much just the model used.