Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

101–110 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#101

Post by ecshafer »

russellbeattie wrote: post #58

I have some macro opinions about Apple - not sure if I'm correct, but tell me what you think. Apple has always seen RAM as an economic advantage for their platform: Make the development effort to ensure that the OS and apps work well with minimal memory and save billions every year in hardware costs. In 2026, iPhones still come with 8Gb of RAM, Pro/Max come with 12Gb. The problem is that AI (ML/LLM training and infer…

In a recent episode of Dwarkesh the guest who is a semiconductor industry analyst predicted that an iPhone will increase in price by about $250 for the same stuff due to increased ram/chip costs from AI. Apple will not be able to afford to put a bunch more RAM into the phones and still sell them.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#102

Post by ezst »

vntok wrote: post #86
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

2 years ago, LLMs failed at answering coherently. Last year, they failed at answering fast on optimized servers. Now, they're failing at answering fast on underpowered handheld devices... I can't wait to see what they'll be failing to do next year.

Probably the one elephant in the roomy thing that matters: failing to say they don't know/can't answer

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#103

Post by superjan »

simonw wrote: post #10
firstbabylonian wrote: post #5

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

Yes. I collected some details here: https://simonwillison.net/2026/Mar/18/llm-in-a-flash/

That was a very good summary. One detail the post could use is mentioning that 4 or 10 experts invoked where selected from the 512 experts the model has per layer (to give an idea of the savings).

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#104

Post by thecopy »

simonw wrote: post #9
cj00 wrote: post #8

It’s 400B but it’s mixture of experts so how many are active at any time?

Looks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App

Stupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB

EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#105

Post by pinkgolem »

Terretta wrote: post #91
zozbot234 wrote: post #53

Earlier quoted context omitted.

If the bottleneck is storage bandwidth that's not "slow". It's only slow if you insist on interactive speeds, but the point of this is that you can run cheap inference in bulk on very low-end hardware.

> very low-end hardware iPhone 17 Pro outperforms AMD’s Ryzen 9 9950X per https://www.igorslab.de/en/iphone-17-pro-a19-pro-chip-uebert...

In single threaded workloads, still impressive

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#106

Post by xg15 »

lesam wrote: post #95
ep103 wrote: post #89

Earlier quoted context omitted.

Some one should let Douglas Adams know the calculation could have been so much faster if the machine just lied.

I think Adams was prescient, since in his story the all powerful computer reaches the answer '42' via incorrect arithmetic.

The Bistromathics? That's not incorrect, it's simply too advanced for us to understand.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#107

Post by AnonymousPlanet »

intrasight wrote: post #28
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

Better than waiting 7.5 million years to have a tell you the answer is 42.

Yes and then no one knows the prompt!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#108

Post by yorwba »

zozbot234 wrote: post #56
snovv_crash wrote: post #31

Earlier quoted context omitted.

The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.

> maybe even learned prefetching for what the next experts will be Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.

It's feasible to put the expert routing logic in a previous layer. People have done it: https://arxiv.org/abs/2507.20984

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#109

Post by eru »

ezst wrote: post #102
vntok wrote: post #86

Earlier quoted context omitted.

2 years ago, LLMs failed at answering coherently. Last year, they failed at answering fast on optimized servers. Now, they're failing at answering fast on underpowered handheld devices... I can't wait to see what they'll be failing to do next year.

Probably the one elephant in the roomy thing that matters: failing to say they don't know/can't answer

With tool use, it's actually quite doable!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#110

Post by post-it »

ezst wrote: post #102
vntok wrote: post #86

Earlier quoted context omitted.

2 years ago, LLMs failed at answering coherently. Last year, they failed at answering fast on optimized servers. Now, they're failing at answering fast on underpowered handheld devices... I can't wait to see what they'll be failing to do next year.

Probably the one elephant in the roomy thing that matters: failing to say they don't know/can't answer

Claude does it all the time, in my experience.
Post reply on HN