Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama3 running locally on iPhone 15 Pro

imgur.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Llama3 running locally on iPhone 15 Pro

#41

Post by davedx »

woadwarrior01 wrote: post #37

Is this news? I've got a nearly year old app that supports over 2 dozen local LLMs with support for using them with Siri and Shortcuts. I added support for Llama 3 8B the day after it came out and also Eric Hartford's new Llama 3 8B based Dolphin model. All models in it are quantized with OmniQuant. On iOS, 7B and 8B ones are 3-bit quantized and smaller models are 4-bit quantized. On the macOS version all models are…

Nice. What is battery life like under heavy use? I was reading a thread on the llama.cpp repo earlier where they were discussing whether it was possible (or attractive) to add neural engine support in some form.

Re: Llama3 running locally on iPhone 15 Pro

#42

Post by hnfong »

nnq wrote: post #31
cqqxo4zV46cp wrote: post #28

Earlier quoted context omitted.

8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?

...so, you haven't used Android in over a decade?

> > we are talking about iPhones here

Re: Llama3 running locally on iPhone 15 Pro

#43

Post by freedomben »

paxys wrote: post #35
aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

Zero chance the marketing department will let them give up the extra $400 or whatever they get to charge for the bare minimum storage and RAM upgrades on all their devices.

I think it's silly to think the marketing department gets to control the pricing, but it is definitely very true that the "starting at " is very powerful for them. Even beyond Apple, it warps and distorts the entire laptop field pricing because people who don't understand how inadequate the entry level model is will compare that price to an entry-level model of Lenovo, or Dell, etc and make conclusions. Even on HN I've seen people use the "starting at" price of macs as a way of "proving" that "the Apple tax isn't much."

So yes, there is tremendous marketing value from that low starting price, although I think it's nearing the end of it's usefulness now that even fan sites are starting to call out the inadequacy.

Re: Llama3 running locally on iPhone 15 Pro

#45

Post by Fubarberry »

__rito__ wrote: post #11

Which app is this? Does anything similar exist for Android?

There's an app called Private AI that will let you run models locally on Android. It has a few smaller models available for free to try it out, but the larger models like Llama 3 (or the option to use your own downloaded models) require a $10 unlock purchase.

Re: Llama3 running locally on iPhone 15 Pro

#46

Post by woadwarrior01 »

davedx wrote: post #41
woadwarrior01 wrote: post #37

Is this news? I've got a nearly year old app that supports over 2 dozen local LLMs with support for using them with Siri and Shortcuts. I added support for Llama 3 8B the day after it came out and also Eric Hartford's new Llama 3 8B based Dolphin model. All models in it are quantized with OmniQuant. On iOS, 7B and 8B ones are 3-bit quantized and smaller models are 4-bit quantized. On the macOS version all models are…

Nice. What is battery life like under heavy use? I was reading a thread on the llama.cpp repo earlier where they were discussing whether it was possible (or attractive) to add neural engine support in some form.

With bigger 7B and 8B models, the battery life goes from a over a day to a few hours on my iPhone 15 Pro.

The 8B model nominally works on 6GB phones but it's quite slow on them. OTOH, it's very usable on iPhone 15 Pro/ Pro Max devices and even better on M1/M2 iPads.

Every framework: llama.cpp, MLX, mlc-llm (which I use) all only use the GPU. Using the ANE and perhaps the undocumented AMX coprocessor for efficient decoder only transformer inference is still an open problem. I've made early some progress on quantised inference using ANE, but there 're still a lot of issues to be solved before it is even demo ready, let alone a shipping product.

Re: Llama3 running locally on iPhone 15 Pro

#47

Post by aeyes »

freedomben wrote: post #43
paxys wrote: post #35

Earlier quoted context omitted.

Zero chance the marketing department will let them give up the extra $400 or whatever they get to charge for the bare minimum storage and RAM upgrades on all their devices.

I think it's silly to think the marketing department gets to control the pricing, but it is definitely very true that the "starting at " is very powerful for them. Even beyond Apple, it warps and distorts the entire laptop field pricing because people who don't understand how inadequate the entry level model is will compare that price to an entry-level model of Lenovo, or Dell, etc and make conclusions. Even on HN I'…

I don't think that they are inadequate, these devices are perfect for most of my family. They do some calls, messages, a couple of pictures here and there, basic word processing and web browsing but not much more on these devices.

I had a Macbook with 8GB RAM and 256GB disk as my daily driver for work until last year running Docker and my fat IDE without too many issues. It's a similar story with my phone - I bought the bigger storage version because I thought I'd need it but after 3 years of using it I'm still not close to even using 128GB.

Re: Llama3 running locally on iPhone 15 Pro

#49

Post by woadwarrior01 »

aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

So the trend with Apple has been that the SoC from the current generation Pro and Pro Max devices becomes the SoC for the next generation of baseline devices. For instance the iPhone 14 Pro Max and iPhone 15 have the same SoC (A16 Bionic). And this trend holds all the way back to iPhone 12.

It's almost certain that the iPhone 16 will ship with 8GB of RAM. What needs to be seen is whether iPhone 16 Pro and Pro Maxes will ship with 16GB of RAM (Like with high end M1/M2 iPad Pros with >= 1TB SSD).

Re: Llama3 running locally on iPhone 15 Pro

#50

Post by ben_w »

nnq wrote: post #31
cqqxo4zV46cp wrote: post #28

Earlier quoted context omitted.

8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?

...so, you haven't used Android in over a decade?

Last time I cared how much RAM any phone had, iOS or Android, I was working at Augmentra on the ViewRanger app, and we were still supporting older devices with only 256 MB.

That was… *checks CV*… I left in April 2015.

I think RAM is like roads: usage expands to fill available infrastructure/storage.

That an iPhone today has as much RAM as the still-functioning Mid-2013 MacBook Air sitting in a drawer behind me is surprising when compared to the 250-fold growth from my Commodore 64 to my (default) Performa 5200… but it doesn't seem to have actually harmed anything I care about.

Post reply on HN