Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

11–20 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#11

Post by pdpi »

cogman10 wrote: post #6
ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

It's both.

We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#14

Post by raw_anon_1111 »

rwaksmunski wrote: post #12

Apple might just win the AI race without even running in it. It's all about the distribution.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store.

It’s only paying Google $1 billion a year for access to Gemini for Siri

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#16

Post by smallerize »

cogman10 wrote: post #6
ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#17

Post by zozbot234 »

firstbabylonian wrote: post #5

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

A similar approach was recently featured here: https://news.ycombinator.com/item?id=47476422 Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#18

Post by [deleted] »

smallerize wrote: post #16
cogman10 wrote: post #6

Earlier quoted context omitted.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).

[deleted]

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#19

Post by detourdog »

raw_anon_1111 wrote: post #14
rwaksmunski wrote: post #12

Apple might just win the AI race without even running in it. It's all about the distribution.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store. It’s only paying Google $1 billion a year for access to Gemini for Siri

Apple’s entire yearly capex is a fraction of the AI spend of the persumed AI winners.
Post reply on HN