Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

31–40 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#31

Post by snovv_crash »

mannyv wrote: post #22
ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

The software has real software engineers working on it instead of researchers. Remember when people were arguing about whether to use mmap? What a ridiculous argument. At some point someone will figure out how to tile the weights and the memory requirements will drop again.

The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#32

Post by Aurornis »

zozbot234 wrote: post #17
firstbabylonian wrote: post #5

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

A similar approach was recently featured here: https://news.ycombinator.com/item?id=47476422 Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)

> Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model.

This is why mixture of experts (MoE) models are favored for these demos: Only a portion of the weights are active for each token.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#33

Post by Aurornis »

causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

I thought you were being sarcastic until I watched the video and saw those words slowly appear.

Emphasis on slowly.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#34

Post by Tade0 »

_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

Only way to have hardware reach this sort of efficiency is to embed the model in hardware.

This exists[0], but the chip in question is physically large and won't fit on a phone.

[0] https://www.anuragk.com/blog/posts/Taalas.html

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#35

Post by WarmWash »

baal80spam wrote: post #30
WarmWash wrote: post #26

Earlier quoted context omitted.

I don't think we are ever going to win this. The general population loves being glazed way too much.

> The general population loves being glazed way too much. This is 100% correct!

Thanks for short warm blast of dopamine, no one else ever seems to grasp how smart I truly am!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#36

Post by originalvichy »

_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

On smartphones? It’s not worth it to run a model this size on a device like this. A smaller fine-tuned model for specific use cases is not only faster, but possibly more accurate when tuned to specific use cases. All those gigs of unnecessary knowledge are useless to perform tasks usually done on smartphones.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#37

Post by foobiekr »

detourdog wrote: post #19
raw_anon_1111 wrote: post #14

Earlier quoted context omitted.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store. It’s only paying Google $1 billion a year for access to Gemini for Siri

Apple’s entire yearly capex is a fraction of the AI spend of the persumed AI winners.

Fantasy buildouts of hundreds of billions of dollars for gear that has a 3 year lifetime may be premature.

Put another way, there is no demonstrated first mover advantage in LLM-based AI so far and all of the companies involved are money furnaces.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#38

Post by Aurornis »

ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5.

It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#40

Post by thinkingtoilet »

intrasight wrote: post #28
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

Better than waiting 7.5 million years to have a tell you the answer is 42.

Maybe you should have asked a better question. :P
Post reply on HN