Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

201–210 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#201

Post by illwrks »

Aachen wrote: post #192
illwrks wrote: post #180

I installed Termux on an old Android phone last week (running LineageOS), and then using Termux installed Ollama and a small model. It ran terribly, but it did run.

Somehow this reminds me of the time I downloaded, compiled, and ran a Bitcoin miner with the app called Linux Deploy on my then-new Galaxy Note (the thing called phablet that is now positively small). It ran terribly, but it did run! Having a complete computer in my pocket was very new to me, coming from Nokia where I struggled (as a teenager) to get any software running besides some JS in a browser. I still don't kn…

Yes, computer in your pocket indeed! I think the Apple Neo shows just how powerful/capable the mobile chips are getting for computer use.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#204

Post by aktau »

xg15 wrote: post #106
lesam wrote: post #95

Earlier quoted context omitted.

I think Adams was prescient, since in his story the all powerful computer reaches the answer '42' via incorrect arithmetic.

The Bistromathics? That's not incorrect, it's simply too advanced for us to understand.

“What do you get if you multiply six by nine?”

(One) source: https://www.reddit.com/r/Fedora/comments/1mjudsm/comment/n7d...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#206

Post by xg15 »

aktau wrote: post #204
xg15 wrote: post #106

Earlier quoted context omitted.

The Bistromathics? That's not incorrect, it's simply too advanced for us to understand.

“What do you get if you multiply six by nine?” (One) source: https://www.reddit.com/r/Fedora/comments/1mjudsm/comment/n7d...

Ok, my Hitchhiker-foo was too weak, thanks!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#207

Post by QuantumNomad_ »

Aurornis wrote: post #139
zozbot234 wrote: post #78

Earlier quoted context omitted.

Yes but most people are still running MoE models with all experts loaded in RAM! This experiment shows quite clearly that some experts are only rarely needed, so you do benefit from not caching every single expert-layer in RAM at all times.

That's not what this test shows. It's just loading the parts of the model that are used in an on-demand fashion from flash. The iPhone 17 Pro only has 12GB of RAM. This is a -17B MoE model. Even quantized, you can only realistically fit one expert in RAM at a time. Maybe 2 with extreme quantization. It's just swapping them out constantly. If some of the experts were unused then you could distill them away. This has b…

If I only use an LLM to ask questions about programming in one specific programming language, can I distill away other experts and get all the answers I need from a single expert? Or is it still different experts that end up handling the question depending on what else is in the question? For example, if I say “plan a static web server in Rust” it might use expert A for that, but if I say “implement a guessing game in Rust” it might use expert B, and so on?

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#208

Post by kgeist »

Aurornis wrote: post #153
thecopy wrote: post #104

Earlier quoted context omitted.

Stupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App

Running larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality. With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most fro…

>I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations.

There are dynamic quants such as Unsloth which quantize only certain layers to Q4. Some layers are more sensitive to quantization than others. Smaller models are more sensitive to quantization than the larger ones. There are also different quantization algorithms, with different levels of degradation. So I think it's somewhat wrong to put "Q4" under one umbrella. It all depends.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#209

Post by Aurornis »

kgeist wrote: post #208
Aurornis wrote: post #153

Earlier quoted context omitted.

Running larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality. With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most fro…

>I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations. There are dynamic quants such as Unsloth which quantize only certain layers to Q4. Some layers are more sensitive to quantization than others. Smaller models are more sensitive to quantization than the larger ones. There are also different quantization algorithms, with different levels of degrada…

I should clarify that I'm referring generically to the types of quantizations used in local LLM inference, including those from Unsloth.

Nobody actually quantizes every layer to Q4 in a Q4 quant.

Post reply on HN