141–150 of 267 posts
My iPhone 13 can’t run most of these models. A decent local LLM is one of the few reasons I can imagine actually upgrading earlier than typically necessary.
Earlier quoted context omitted.
I haven't seen anybody else post it in this thread, but this is running on 8GB of RAM. It's not the full Gemma 4 32B model. It's a completely different thing from the full Gemma 4 experience if you were running the flagship model, almost to the point of being misleading. It's their E2B and E4B variants (so 2B and 4B but also quantized) https://ai.google.dev/gemma/docs/core/model_card_4#dense_mod...
The relevant constraint when running on a phone is power, not really RAM footprint. Running the tiny E2B/E4B models makes sense, this is essentially what they're designed for.
So much so that this was what made Apple increase their base sizes.
This app is cool and it showcases some use cases, but it still undersells what the E2B model can do. I just made a real-time AI (audio/video in, voice out) on an M3 Pro with Gemma E2B. I posted it on /r/LocalLLaMA a few hours ago and it's gaining some traction [0]. Here's the repo [1] I'm running it on a Macbook instead of an iPhone, but based on the benchmark here [2], you should be able to run the same thing on an…
Saw this one on X the other day updated with Gemma 4 and they have the built-in Apple Foundation model, Qwen3.5, and other models:
Locally AI - https://locallyai.app/
English version of the page: https://apps.apple.com/us/app/google-ai-edge-gallery/id67496... Also on Android: https://play.google.com/store/apps/details?id=com.google.ai.... It's a demo app for Google's Edge project: https://ai.google.dev/edge
Earlier quoted context omitted.
> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…
> It's widely understood that the big players are making profit on inference. If you add in the cost of training, it’s not profitable. Not including the cost of training is a bit like saying the only cost of a cup of coffee is the paper cup it’s in. The only way OpenAI gets to charge for inference is by selling a product people can’t get elsewhere for much cheaper, which means billions in R&D costs. But because of co…
Earlier quoted context omitted.
Did you really watch “Her” and think this is a future that should happen?? Seriously????
What does what they said have anything to do with Her? Local LLMs are better than big corporations owning your data and offering LLMs for a huge cost.
English version of the page: https://apps.apple.com/us/app/google-ai-edge-gallery/id67496... Also on Android: https://play.google.com/store/apps/details?id=com.google.ai.... It's a demo app for Google's Edge project: https://ai.google.dev/edge
Gemma4 works really slow on my android e2b model on Samsung galaxy s21 ultra. Atleast 20-30 sec to warm up and then reply.
OP Here. It is my firm belief that the only realistic use of AI in the future is either locally on-device for almost free, or in the cloud but way more expensive then it is today. The latter option will only bemusedly for tasks that humans are more expensive or much slower in. This Gemma 4 model gives me hope for a future Siri or other with iPhone and macOS integration, “Her” (as in the movie) style.
> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…
Vast amounts of capital have been poured in, but they continue to raise more. Presumably because they need more.
Is the capital being invested without any expectation of ROI?