[dead]
iPhone 17 Pro Demonstrated Running a 400B LLM
151–160 of 362 posts
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#152Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. I understand this is for a demo but do we really need a 400B model in the mobile? A 10B model would do fine right? What do we miss with a pared down one?
What do we miss?
Tl;dr a lot, model is much worse
(Source: maintaining llama.cpp / cloud based llm provider app for 2-3 years now)
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#153Earlier quoted context omitted.
Looks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App
Stupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App
Running larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality.
With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#154[flagged]
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#155A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.
/FIFY A year ago this would have been considered impossible. The software is moving faster than anyone's hardware assumptions.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#156Earlier quoted context omitted.
It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5. It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.
If the bottleneck is storage bandwidth that's not "slow". It's only slow if you insist on interactive speeds, but the point of this is that you can run cheap inference in bulk on very low-end hardware.
> If the bottleneck is storage bandwidth that's not "slow"
It is objectively slow at around 100X slower than what most people consider usable.
The quality is also degraded severely to get that speed.
> but the point of this is that you can run cheap inference in bulk on very low-end hardware.
You always could, if you didn't care about speed or efficiency.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#157"400 bytes should be enough for anybody"
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#158I can't understand why this is a surprise to anyone. An iphone is still a computer, of course it can run any model that fits in storage albiet very slowly. The implementation is impressive I guess but I don't see how this is a novel capability. And for 0.6t/s, its not a cost efficient hardware for doing it. The iphone can also render pixar movies if you let it run long enough, mine bitcoin with a pathetic hashrate, and do weather simulations but not in time for the forecast to be relevant.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#159Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. I understand this is for a demo but do we really need a 400B model in the mobile? A 10B model would do fine right? What do we miss with a pared down one?
> Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design. Mobile phones don't have separate GPUs and separate VRAM like some desktops. This isn't a new thing and it's not uniqu…
> Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design.
> Mobile phones don't have separate GPUs and separate VRAM like some desktops.
That's true. The difference is the iPhone has wider memory buses and uses faster LPDDR5 memory. Apple places the RAM dies directly on the same package as the SoC (PoP — Package on Package), minimizing latency. Some Android phones have started to do this, too.
iOS is tuned to this architecture which wouldn't be the case across many different Android hardware configurations.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#160"400 bytes should be enough for anybody"
The 'B' in 400B is billion, not bytes. And the quote '640k ought to be enough for everyone' doesn't have evidence supporting Bill G said it: https://www.computerworld.com/article/1563853/the-640k-quote....
That said, it'd be a fun quote and I've jokingly said it as well, as I think of it more as part of 'popular' culture lol