Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

51–60 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#51

Post by intrasight »

Tade0 wrote: post #34
_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html

I think for many reasons this will become the dominant paradigm for end user devices.

Moore's law will shrink it to 8mm soon. I think it'll be like a microSD card you plug in.

Or we develop a new silicon process that can mimic synaptic weights in biology. Synapses have plasticity.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#52

Post by bigyabai »

pdpi wrote: post #11
cogman10 wrote: post #6

Earlier quoted context omitted.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

It's both. We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.

> We haven't had phones running laptop-grade CPUs/GPUs for that long

Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#53

Post by zozbot234 »

Aurornis wrote: post #38
ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5. It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.

If the bottleneck is storage bandwidth that's not "slow". It's only slow if you insist on interactive speeds, but the point of this is that you can run cheap inference in bulk on very low-end hardware.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#55

Post by Multiplayer »

icedchai wrote: post #41
qingcharles wrote: post #25

Earlier quoted context omitted.

Plus all those pricey 512GB Mac Studios they are selling to YouTubers.

They don't offer the 512 gig RAM variant anymore. Outside of social media influencers and the occasional AI researcher, the market for $10K desktops is vanishingly small.

My understanding is that the 512gb offering will likely return with the new M5 Ultra coming around WWDC in June. Fingers crossed anyway!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#56

Post by zozbot234 »

snovv_crash wrote: post #31
mannyv wrote: post #22

Earlier quoted context omitted.

The software has real software engineers working on it instead of researchers. Remember when people were arguing about whether to use mmap? What a ridiculous argument. At some point someone will figure out how to tile the weights and the memory requirements will drop again.

The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.

> maybe even learned prefetching for what the next experts will be

Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#57

Post by 9dev »

WarmWash wrote: post #26
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

I don't think we are ever going to win this. The general population loves being glazed way too much.

You’re absolutely right!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#58

Post by russellbeattie »

I have some macro opinions about Apple - not sure if I'm correct, but tell me what you think.

Apple has always seen RAM as an economic advantage for their platform: Make the development effort to ensure that the OS and apps work well with minimal memory and save billions every year in hardware costs. In 2026, iPhones still come with 8Gb of RAM, Pro/Max come with 12Gb.

The problem is that AI (ML/LLM training and inference) are areas where you can't get around the need for copious amounts of fast working memory. (Thus the critical shortage of RAM at the moment as AI data centers consume as many memory chips as possible.)

Unless there's something I don't know (which is more than possible) Apple can't code their way around this problem, nor create specialized SoCs with ML cores that obviate the need for lots and lots of RAM.

So, it's going to be interesting whether they accept this reality and we start seeing the iPhones in the future with 16Gb, 32Gb or more as standard in order to make AI performant. And if they give up on adding AI to the billions of iPhones with minimal RAM already out there.

As a side note, 8Gb of RAM hasn't been enough for a decade. It prevents basic tasks like keeping web tabs live in the background. My pet peeve is having just a few websites open, and having the page refresh when swapping between them because of aggressive memory management.

To me, Apple's obvious strength is pushing AI to the edge as much as possible. While other companies are investing in massive data centers which will have millions of chips that will be outdated within the next couple years, Apple will be able to incrementally improve their ML/AI features by running on the latest and greatest chips every year. Apple has a huge advantage in that they can design their chips with a mega high speed bus, which is just as important as the quantity of RAM.

But all that depends on Apple's willingness to accept that RAM isn't an area they can skimp on any more, and I'm not sure they will.

Sorry for the brain dump. I'd love to be educated on this in case I'm totally off base.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#59

Post by bigyabai »

intrasight wrote: post #51
Tade0 wrote: post #34

Earlier quoted context omitted.

Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html

I think for many reasons this will become the dominant paradigm for end user devices. Moore's law will shrink it to 8mm soon. I think it'll be like a microSD card you plug in. Or we develop a new silicon process that can mimic synaptic weights in biology. Synapses have plasticity.

One big bottleneck is SRAM cost. Even an 8b model would probably end up being hundreds of dollars to run locally on that kind of hardware. Especially unpalatable if the model quality keeps advancing year-by-year.

> Or we develop a new silicon process that can mimic synaptic weights in biology. Synapses have plasticity.

It's amazing to me that people consider this to be more realistic than FAANG collaborating on a CUDA-killer. I guess Nvidia really does deserve their valuation.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#60

Post by snovv_crash »

zozbot234 wrote: post #56
snovv_crash wrote: post #31

Earlier quoted context omitted.

The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.

> maybe even learned prefetching for what the next experts will be Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.

Manually no. It would have to be learned, and making the expert selection predictable would need to be a training metric to minimize.
Post reply on HN