[dead]
iPhone 17 Pro Demonstrated Running a 400B LLM
141–150 of 362 posts
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#142The power draw is going to be crazy (today).
Practical LLMs on mobile devices are at least a few years away.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#143My iPad Air with M2 can run local LLMs rather well. But it gets ridiculously hot within seconds and starts throttling.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#144Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."
I don't think we are ever going to win this. The general population loves being glazed way too much.
Poor “we”. “They” love looking at their own reflection too much.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#145Earlier quoted context omitted.
Looks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App
Stupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App
I have a 64G/1T Studio with an M1 Ultra. You can probably run this model to say you’ve done it but it wouldn’t be very practical.
Also I wouldn’t trust 3-bit quantization for anything real. I run a 5-bit qwen3.5-35b-A3B MoE model on my studio for coding tasks and even the 4-bit quant was more flaky (hallucinations, and sometimes it would think about running tools calls and just not run them, lol).
If you decided to give it a go make sure to use the MLX over the GGUF version! You’ll get a bit more speed out of it.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#146Earlier quoted context omitted.
That's actually pretty cool, but I'd hate to freeze a models weights into silicon without having an incredibly specific and broad usecase.
Sounds like just the sort of thing FGPA's were made for. The $$$ would probably make my eyes bleed tho.
Current FPGAs would have terrible performance. We need some new architecture combining ASIC LLM perf and sparse reconfiguration support maybe.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#147[dead]
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#148Earlier quoted context omitted.
Yes but most people are still running MoE models with all experts loaded in RAM! This experiment shows quite clearly that some experts are only rarely needed, so you do benefit from not caching every single expert-layer in RAM at all times.
That's not what this test shows. It's just loading the parts of the model that are used in an on-demand fashion from flash. The iPhone 17 Pro only has 12GB of RAM. This is a -17B MoE model. Even quantized, you can only realistically fit one expert in RAM at a time. Maybe 2 with extreme quantization. It's just swapping them out constantly. If some of the experts were unused then you could distill them away. This has b…
The writeup from the earlier experiment (running on a MacBook Pro) shows quite clearly that expert routing choices are far from uniform, and that some layer-experts are only used rarely. So you can save some RAM footprint even while swapping quite rarely.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#149Earlier quoted context omitted.
That's not what this test shows. It's just loading the parts of the model that are used in an on-demand fashion from flash. The iPhone 17 Pro only has 12GB of RAM. This is a -17B MoE model. Even quantized, you can only realistically fit one expert in RAM at a time. Maybe 2 with extreme quantization. It's just swapping them out constantly. If some of the experts were unused then you could distill them away. This has b…
The writeup from the earlier experiment (running on a MacBook Pro) shows quite clearly that expert routing choices are far from uniform, and that some layer-experts are only used rarely. So you can save some RAM footprint even while swapping quite rarely.
I understand, but this isn't just a matter of not caching some experts. This is a 397B model on a device with 12GB of RAM. It's basically swapping experts out all the time, even if the distribution isn't uniform.
When the individual expert sizes are similar to the entire size of the RAM on the device, that's your only option.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#150Earlier quoted context omitted.
Better than waiting 7.5 million years to have a tell you the answer is 42.
Looked at a certain way it's incredible that a 40-odd year old comedy sci-fi series is so accurate about the expected quality of (at least some) AI output. Which makes it even funnier. It makes me a little sad that Douglas Adams didn't live to see it.
42 wasn't a low quality answer.
The joke revolves around the incongruity of "42" being precisely correct.