Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

161–170 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#161

Post by zozbot234 »

Aurornis wrote: post #149
zozbot234 wrote: post #148

Earlier quoted context omitted.

The writeup from the earlier experiment (running on a MacBook Pro) shows quite clearly that expert routing choices are far from uniform, and that some layer-experts are only used rarely. So you can save some RAM footprint even while swapping quite rarely.

I understand, but this isn't just a matter of not caching some experts. This is a 397B model on a device with 12GB of RAM. It's basically swapping experts out all the time, even if the distribution isn't uniform. When the individual expert sizes are similar to the entire size of the RAM on the device, that's your only option.

"Individual experts" is a bit of a red-herring, what matters is expert-layers (this is the granularity of routing decisions), and these are small as mentioned by the original writeup. The filesystem cache does a tolerable job of keeping the "often used" ones around while evicting those that aren't needed (this is what their "Trust the OS" point is about). Of course they're also reducing the amount of active experts and quantizing a lot, AIUI this iPhone experiment uses Q1 and the MacBook was Q2.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#163

Post by Aurornis »

alwillis wrote: post #159
Aurornis wrote: post #135

Earlier quoted context omitted.

> Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design. Mobile phones don't have separate GPUs and separate VRAM like some desktops. This isn't a new thing and it's not uniqu…

> Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design. > Mobile phones don't have separate GPUs and separate VRAM like some desktops. That's true. The difference is the iPhone has wider memory buses and uses faster LPDDR5 memory. Apple places the RAM dies directly on the same package as the SoC (PoP — Package on Package), minimizing latency. Some Android…

> The difference is the iPhone has wider memory buses and uses faster LPDDR5 memory. Apple places the RAM dies directly on the same package as the SoC (PoP — Package on Package), minimizing latency. Some Android phones have started to do this, too.

Package-on-Package has been used in mobile SoCs for a long time. This wasn't an Apple invention. It's not new, either. It's been this way for 10+ years. Even cheap Raspberry Pi models have used package-on-package memory.

The memory bandwidth of flagship iPhone models is similar to the memory bandwidth of flagship Android phones.

There's nothing uniquely Apple in this. This is just how mobile SoCs have been designed for a long time.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#164

Post by root_axis »

yalogin wrote: post #132

Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. I understand this is for a demo but do we really need a 400B model in the mobile? A 10B model would do fine right? What do we miss with a pared down one?

Compared to a 400b model, a 10b is practically useless, it's not even worth bothering outside of tinkering for fun and research.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#166

Post by root_axis »

_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

It will never be possible on a smart phone. I know that sounds cynical, but there's basically no path to making this possible from an engineering perspective.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#167

Post by atultw »

bigyabai wrote: post #84
zozbot234 wrote: post #82

Earlier quoted context omitted.

CUDA is not the real issue, AMD's HIP offers source-level compatibility with CUDA code, and ZLUDA even provides raw binary compatibility. nVidia GPUs really are quite good, and the projected advantages of going multi-vendor just aren't worth the hassle given the amount of architecture-specificity GPUs are going to have.

Okay, then don't kill CUDA, just sign CUDA drivers on macOS instead and quit pretending like MPS is a world-class solution. There are trillions on the table, this is not an unsolvable issue.

Admittedly, my use of CUDA and Metal is fairly surface-level. But I have had great success using LLMs to convert whole gaussian splatting CUDA codebases to Metal. It's not ideal for maintainability and not 1:1, but if CUDA was a moat for NVIDIA, I believe LLMs have dealt a blow to it.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#168

Post by zozbot234 »

Aurornis wrote: post #156
zozbot234 wrote: post #53

Earlier quoted context omitted.

If the bottleneck is storage bandwidth that's not "slow". It's only slow if you insist on interactive speeds, but the point of this is that you can run cheap inference in bulk on very low-end hardware.

> If the bottleneck is storage bandwidth that's not "slow" It is objectively slow at around 100X slower than what most people consider usable. The quality is also degraded severely to get that speed. > but the point of this is that you can run cheap inference in bulk on very low-end hardware. You always could, if you didn't care about speed or efficiency.

You're simply pointing out that most people who use AI today expect interactive speeds. You're right that the point here is not raw power efficiency (having to read from storage will impact energy per operation, and datacenter-scale AI hardware beats edge hardware anyway by that metric) but the ability to repurpose cheaper, lesser-scale hardware is also compelling.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#169

Post by tren_hard »

tclancy wrote: post #62
Tade0 wrote: post #34

Earlier quoted context omitted.

Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html

I think you're ignoring the inevitable march of progress. Phones will get big enough to hold it soon.

Instead of slapping on an extra battery pack, it will be an onboard llm model. Could have lifecycles just like phones.

Getting bigger (foldable) phones, without losing battery life, and running useable models in the same form-factor is a pretty big ask.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#170

Post by freedomben »

Aurornis wrote: post #153
thecopy wrote: post #104

Earlier quoted context omitted.

Stupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App

Running larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality. With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most fro…

I've tried a number of experiments, and agree completely. If it doesn't fit in RAM, it's so slow as to be impractical and almost useless. If you're running things overnight, then maybe, but expect to wait a very long time for any answers.
Post reply on HN