Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

21–30 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#21

Post by simonw »

zozbot234 wrote: post #17
firstbabylonian wrote: post #5

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

A similar approach was recently featured here: https://news.ycombinator.com/item?id=47476422 Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)

Yeah, this new post is a continuation of that work.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#22

Post by mannyv »

ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

The software has real software engineers working on it instead of researchers.

Remember when people were arguing about whether to use mmap? What a ridiculous argument.

At some point someone will figure out how to tile the weights and the memory requirements will drop again.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#25

Post by qingcharles »

raw_anon_1111 wrote: post #14
rwaksmunski wrote: post #12

Apple might just win the AI race without even running in it. It's all about the distribution.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store. It’s only paying Google $1 billion a year for access to Gemini for Siri

Plus all those pricey 512GB Mac Studios they are selling to YouTubers.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#26

Post by WarmWash »

causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

I don't think we are ever going to win this. The general population loves being glazed way too much.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#27

Post by devmor »

detourdog wrote: post #19
raw_anon_1111 wrote: post #14

Earlier quoted context omitted.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store. It’s only paying Google $1 billion a year for access to Gemini for Siri

Apple’s entire yearly capex is a fraction of the AI spend of the persumed AI winners.

Which is mostly insane amounts of debt leveraged entirely on the moonshot that they will find a way to turn a profit on it within the next couple years.

Apple’s bet is intelligent, the “presumed winners” are hedging our economic stability on a miracle, like a shaking gambling addict at a horse race who just withdrew his rent money.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#28

Post by intrasight »

causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

Better than waiting 7.5 million years to have a tell you the answer is 42.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#29

Post by naikrovek »

dzikimarian wrote: post #24
rwaksmunski wrote: post #12

Apple might just win the AI race without even running in it. It's all about the distribution.

Because someone managed to run LLM on an iPhone at unusable speed Apple won AI race? Yeah, sure.

whoa, save some disbelief for later, don't show it all at once.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#30

Post by baal80spam »

WarmWash wrote: post #26
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

I don't think we are ever going to win this. The general population loves being glazed way too much.

> The general population loves being glazed way too much.

This is 100% correct!

Post reply on HN