Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama3 running locally on iPhone 15 Pro

imgur.com

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Llama3 running locally on iPhone 15 Pro

#23

Post by mariuz »

Related with GitHub link

"Next level: QLoRA fine-tuning 4-bit Llama 3 8B on iPhone 15 pro.

Incoming (Q)LoRA MLX Swift example by David Koski: https://github.com/ml-explore/mlx-swift-examples/pull/46 works with lot's of models (Mistral, Gemma, Phi-2, etc)"

https://twitter.com/awnihannun/status/1782436898285527229

Re: Llama3 running locally on iPhone 15 Pro

#24

Post by MPSimmons »

aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

I can't wait until Groq or someone else release tiny mobile inference engines specifically for phones and the like.

Re: Llama3 running locally on iPhone 15 Pro

#25

Post by regularfry »

gcanyon wrote: post #18
TMWNN wrote: post #7

Earlier quoted context omitted.

My current and previous MacBooks have had 16GB and I've been fine with it, but given local models I think I'm going to have to go to whatever will be the maximum RAM available for the next one. It runs 13b models quite well with Ollama, but I tried `mixtral-8x7b` and saw 0.25 tokens/second speeds; I suppose I should be amazed that it ran at all. Similarly, I am for the first time going to care about how much RAM is i…

I recently upgraded from my M1 Air specifically because I had purchased it with 8gb -- silly me. Now I have 24gb, and if the Air line had more available I would have sprung for 32, or even 64gb. But I'm not paying for a faster processor just to get more memory :-/

I got an 8GB M1 from work, and I've been frankly astonished with what even this machine can do. Yes, it'll run the 4bit llama3 quants - not especially fast, mind, but not unusably slow either. The problem is that you can't do a huge amount else.

Re: Llama3 running locally on iPhone 15 Pro

#26

Post by MyFirstSass »

aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

It's one of the most infuriating things about Apple.

The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have low ram, ie. no new OS LLM updates.

Also we can't have gaming because half their new sold devices have shit ram, so they also kind of already ditched their "gaming" plan they just got started on a year ago - all because they wan't to push products with ram levels from 10 years ago - bizarre!

They must be betting on local AI as a "pro" feature only.

Re: Llama3 running locally on iPhone 15 Pro

#28

Post by cqqxo4zV46cp »

MyFirstSass wrote: post #26
aNoob7000 wrote: post #22

I wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.

It's one of the most infuriating things about Apple. The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have…

8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic

Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?

Post reply on HN