Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama3 running locally on iPhone 15 Pro

imgur.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Llama3 running locally on iPhone 15 Pro

#31

Post by nnq »

cqqxo4zV46cp wrote: post #28
MyFirstSass wrote: post #26

Earlier quoted context omitted.

It's one of the most infuriating things about Apple. The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have…

8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?

...so, you haven't used Android in over a decade?

Re: Llama3 running locally on iPhone 15 Pro

#32

Post by HumblyTossed »

aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

> They've been stingy on increasing RAM

... in any of their products.

FTFY

Re: Llama3 running locally on iPhone 15 Pro

#33

Post by whywhywhywhy »

MyFirstSass wrote: post #26
aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

It's one of the most infuriating things about Apple. The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have…

8GB ram models don't exist to be used they exist to be e-waste that gets you to the checkout page where you click 16GB instead.

Re: Llama3 running locally on iPhone 15 Pro

#34

Post by iAkashPaul »

dTal wrote: post #19
__rito__ wrote: post #11

Which app is this? Does anything similar exist for Android?

On Android you can simply run vanilla llama.cpp inside a terminal, or indeed any stack that you would run on a Linux desktop that doesn't involve a native GUI.

Yep, termux is a good way to do this. Llama.cpp has Android example as well, I forked it here GitHub.com/iakashpaul/portal you can try it with any supported GGUF/Q4+Q8 models

Re: Llama3 running locally on iPhone 15 Pro

#35

Post by paxys »

aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

Zero chance the marketing department will let them give up the extra $400 or whatever they get to charge for the bare minimum storage and RAM upgrades on all their devices.

Re: Llama3 running locally on iPhone 15 Pro

#37

Post by woadwarrior01 »

Is this news? I've got a nearly year old app that supports over 2 dozen local LLMs with support for using them with Siri and Shortcuts. I added support for Llama 3 8B the day after it came out and also Eric Hartford's new Llama 3 8B based Dolphin model. All models in it are quantized with OmniQuant. On iOS, 7B and 8B ones are 3-bit quantized and smaller models are 4-bit quantized. On the macOS version all models are 4-bit OmniQuant quantized. 3-bit Omniquant quantization is quite comparable in perplexity to 4-bit RTN quantization that all the llama.cpp based apps use.

https://privatellm.app/

https://apps.apple.com/app/private-llm-local-ai-chatbot/id64...

Re: Llama3 running locally on iPhone 15 Pro

#38

Post by sestep »

yohannparis wrote: post #20

This is quite impressive to be honest. The chat is answering at a speed of one word per few/several seconds. But still, this a nice feat. Example recording for the curious: https://www.youtube.com/watch?v=nZEvUj-QTrI

I'm so spoiled by the quality of modern closed-source models, I laughed out loud when the beginning of its answer just said [duplicate].

Re: Llama3 running locally on iPhone 15 Pro

#39

Post by foobarian »

cqqxo4zV46cp wrote: post #28
MyFirstSass wrote: post #26

Earlier quoted context omitted.

It's one of the most infuriating things about Apple. The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have…

8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?

> But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic

Honestly I'm glad they don't. The PC is the last open platform out there and the last thing I'd want to see is Apple encroaching on it with their walled gardens and carbonite-encased computers.

Post reply on HN