Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 on iPhone

apps.apple.com

Post reply on HN
Print view View original

51–60 of 267 posts

Re: Gemma 4 on iPhone

#51

Post by zozbot234 »

crazygringo wrote: post #44
janandonly wrote: post #28

OP Here. It is my firm belief that the only realistic use of AI in the future is either locally on-device for almost free, or in the cloud but way more expensive then it is today. The latter option will only bemusedly for tasks that humans are more expensive or much slower in. This Gemma 4 model gives me hope for a future Siri or other with iPhone and macOS integration, “Her” (as in the movie) style.

> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…

The big players are plausibly making profits on raw API calls, not subscriptions. These are quite costly compared to third-party inference from open models, but even setting that up is a hassle and you as a end user aren't getting any subsidy. Running inference locally will make a lot of sense for most light and casual users once the subsidies for subscription access cease.

Also while datacenter-based scaleout of a model over multiple GPUs running large batches is more energy efficient, it ultimately creates a single point of failure you may wish to avoid.

Re: Gemma 4 on iPhone

#53

Post by zozbot234 »

nothinkjustai wrote: post #46
crazygringo wrote: post #44

Earlier quoted context omitted.

> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…

> It's widely understood that the big players are making profit on inference. Are they? Or are they just saying that to make their offerings more attractive to investors? Plus I think most people using agents for coding are using subscriptions which they are definitely not profitable in. Locally running models that are snappy and mostly as capable as current sota models would be a dream. No internet connection requir…

You can pick models that are snappy, or models that are as capable as SOTA. You don't really get both unless you spend extremely unreasonable amounts of money on what is essentially a datacenter-scale inference platform of your own, meant to service hundreds of users at once. (I don't care how many agent harnesses you spin up at once, you aren't going to get the same utilization as hundreds of concurrent users.)

This assessment might change if local AI frameworks start working seriously on support for tensor-parallel distributed inference, then you might get away with cheaper homelab-class hardware and only mildly unreasonable amounts of money.

Re: Gemma 4 on iPhone

#55

Post by amelius »

janandonly wrote: post #28

OP Here. It is my firm belief that the only realistic use of AI in the future is either locally on-device for almost free, or in the cloud but way more expensive then it is today. The latter option will only bemusedly for tasks that humans are more expensive or much slower in. This Gemma 4 model gives me hope for a future Siri or other with iPhone and macOS integration, “Her” (as in the movie) style.

A local model running on a phone owned and controlled by the vendor is still not really exciting, imho.

It may be physically "local" but not in spirit.

Re: Gemma 4 on iPhone

#57

Post by mbesto »

crazygringo wrote: post #44
janandonly wrote: post #28

OP Here. It is my firm belief that the only realistic use of AI in the future is either locally on-device for almost free, or in the cloud but way more expensive then it is today. The latter option will only bemusedly for tasks that humans are more expensive or much slower in. This Gemma 4 model gives me hope for a future Siri or other with iPhone and macOS integration, “Her” (as in the movie) style.

> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…

> It's widely understood that the big players are making profit on inference.

This is most definitely not widely understood. We still don't know yet. There's tons of discussions about people disagreeing on whether it really is profitable. Unless you have proof, don't say "this is widely understood".

Re: Gemma 4 on iPhone

#58

Post by huijzer »

crazygringo wrote: post #44
janandonly wrote: post #28

OP Here. It is my firm belief that the only realistic use of AI in the future is either locally on-device for almost free, or in the cloud but way more expensive then it is today. The latter option will only bemusedly for tasks that humans are more expensive or much slower in. This Gemma 4 model gives me hope for a future Siri or other with iPhone and macOS integration, “Her” (as in the movie) style.

> or in the cloud but way more expensive then it is today. Why? It's widely understood that the big players are making profit on inference. The only reason they still have losses is because training is so expensive, but you need to do that no matter whether the models are running in the cloud or on your device. If you think about it, it's always going to be cheaper and more energy-efficient to have dedicated cloud ha…

Laptop/desktop could work. Most systems are on charger most of time anyway

Re: Gemma 4 on iPhone

#59

Post by rav3ndust »

throwuxiytayq wrote: post #26
barbazoo wrote: post #23

Earlier quoted context omitted.

> And there's a whole set of ethically-justifiable but rule-flagging conversations (loosely categorizable as things like "sensitive", "ethically-borderline-but-productive" or "violating sacred cows") that are now possible with this, and at a level never before possible until now. I checked the abliterate script and I don't yet understand what it does or what the result is. What are the conversations this enables?

The in-ter-net is for porn

that song is going to be stuck in my head all day now. lol

Re: Gemma 4 on iPhone

#60

Post by karimf »

nothinkjustai wrote: post #49
karimf wrote: post #40

This app is cool and it showcases some use cases, but it still undersells what the E2B model can do. I just made a real-time AI (audio/video in, voice out) on an M3 Pro with Gemma E2B. I posted it on /r/LocalLLaMA a few hours ago and it's gaining some traction [0]. Here's the repo [1] I'm running it on a Macbook instead of an iPhone, but based on the benchmark here [2], you should be able to run the same thing on an…

Parlor is so cool, especially since you’re offering it for free. And a great use case for local LLMs.

Thanks! Although, I can't claim any credit for it. I just spent a day gluing what other people have built. Huge props to the Gemma team for building an amazing model and also an inference engine that's focused for edge devices [0]

[0] https://github.com/google-ai-edge/LiteRT-LM

Post reply on HN