iPhone 17 Pro Demonstrated Running a 400B LLM

It's crazy to see a 400B model running on an iPhone. But moving forward, as the information density and architectural efficiency of smaller models continue to increase, getting high-quality, real-time inference on mobile is going to become trivial.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#5

Post by firstbabylonian » Mon, Mar 23, 2026, 3:01 PM UTC

> SSD streaming to GPU

Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]?

1: https://arxiv.org/abs/2312.11514

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#6

Post by cogman10 » Mon, Mar 23, 2026, 3:01 PM UTC

ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

This isn't a hardware feat, this is a software triumph.

They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#7

Post by lostmsu » Mon, Mar 23, 2026, 3:02 PM UTC

anemll wrote: post #2

[flagged]

This has nothing to do with Apple, and everything to do with MoE and that everyone forgot you can re-read the necessary bits of the model from disk for each token.

This is extremely inefficient though. For efficiency you need to batch many requests (like 32+, probably more like 128+), and when you do that with MoE you lose the advantage of only having to read a subset of the model during a single forward pass, so the trick does not work.

But this did remind me that with dense models you might be able to use disk to achieve high throughput at high latency on GPUs that don't have a lot of VRAM.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#8

Post by cj00 » Mon, Mar 23, 2026, 3:08 PM UTC

It’s 400B but it’s mixture of experts so how many are active at any time?

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#9

Post by simonw » Mon, Mar 23, 2026, 3:10 PM UTC

cj00 wrote: post #8

It’s 400B but it’s mixture of experts so how many are active at any time?

Looks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#10

Post by simonw » Mon, Mar 23, 2026, 3:10 PM UTC

firstbabylonian wrote: post #5

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

Yes. I collected some details here: https://simonwillison.net/2026/Mar/18/llm-in-a-flash/