It will be funny if we go back to lugging around brick-size batteries with us everywhere!
iPhone 17 Pro Demonstrated Running a 400B LLM
111–120 of 362 posts
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#112Earlier quoted context omitted.
It's both. We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.
> We haven't had phones running laptop-grade CPUs/GPUs for that long Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.
Kind of.
We've had solid CPUs for a while, but GPUs have lagged behind (and they're the ones that matter for this particular application). iPhones still lead by a comfortable margin on this front, but have historically been pretty limited on the IO front (only supported USB2 speeds until recently).
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#113This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
Is 100 t/s the stadard for models?
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#114Impressive. Running a 400B model on-device, even at low throughput, is pretty wild.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#115[dead]
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#116Earlier quoted context omitted.
Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html
I think you're ignoring the inevitable march of progress. Phones will get big enough to hold it soon.
I think the future is the model becoming lighter not the hardware becoming heavier
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#117I have some macro opinions about Apple - not sure if I'm correct, but tell me what you think. Apple has always seen RAM as an economic advantage for their platform: Make the development effort to ensure that the OS and apps work well with minimal memory and save billions every year in hardware costs. In 2026, iPhones still come with 8Gb of RAM, Pro/Max come with 12Gb. The problem is that AI (ML/LLM training and infer…
I think this is roughly true, but instead RAM will remain a discriminator even moreso. If the scaling laws apple has domain over are compute and model size, then they'll pretty easily be able to map that into their existing price tiers.
Pros will want higher intelligence or throughput. Less demanding or knowledgeable customers will get price-funneled to what Apple thinks is the market premium for their use case.
It'll probably be a little harder to keep their developers RAM disciplined (if that's even still true) for typical concerns. But model swap will be a big deal. The same exit vs voice issues will exist for apple customers but the margin logic seems to remain.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#118Earlier quoted context omitted.
Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html
That's actually pretty cool, but I'd hate to freeze a models weights into silicon without having an incredibly specific and broad usecase.
Depends on cost IMO - if I could buy a Kimi K2.5 chip for a couple of hundred dollars today I would probably do it.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#119Earlier quoted context omitted.
Better than waiting 7.5 million years to have a tell you the answer is 42.
Looked at a certain way it's incredible that a 40-odd year old comedy sci-fi series is so accurate about the expected quality of (at least some) AI output. Which makes it even funnier. It makes me a little sad that Douglas Adams didn't live to see it.
Also check out "The Great Automatic Grammatizator" by Roald Dahl for another eerily accurate scifi description of LLMs written in 1954:
https://gwern.net/doc/fiction/science-fiction/1953-dahl-theg...