Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

241–250 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#241

Post by andyferris »

echelon wrote: post #240
aetherspawn wrote: post #237

Earlier quoted context omitted.

Is it though? I would say 'proof of concept' instead. The fact that it's running on a phone now just sets the goalpost and gets everyone excited about it: add more RAM and GPU to the next iPhone and it's not a toy anymore. Co-incidentally, phone companies also have thousands of engineers sitting around wondering what to do in their next release to convince consumers to buy ...

> add more RAM and GPU to the next iPhone and it's not a toy anymore We're not going to get more RAM and GPU in consumer devices. All of the supply is going into data center build outs. As the hyper scaler gamble on the future continues, we get left with weaker (or more expensive) devices - not stronger ones. The market makers make more money if we're left to thin clients. They're also the ones who control supply and…

I highly doubt the A20 Pro will be slower than the A19 Pro - particularly for AI workloads.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#242

Post by bigyabai »

anemll wrote: post #173
skiing_crawling wrote: post #158

I can't understand why this is a surprise to anyone. An iphone is still a computer, of course it can run any model that fits in storage albiet very slowly. The implementation is impressive I guess but I don't see how this is a novel capability. And for 0.6t/s, its not a cost efficient hardware for doing it. The iphone can also render pixar movies if you let it run long enough, mine bitcoin with a pathetic hashrate, a…

SSD streaming to compute units is new. M4 max can do 15 t/s with its 15GB/s drives

It was "new" in 2019. The PS5 and Xbox Series X both shipped with GPUDirect Storage, and even most dGPUs support it via ReBAR/RDMA nowadays.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#243

Post by bigyabai »

atultw wrote: post #167
bigyabai wrote: post #84

Earlier quoted context omitted.

Okay, then don't kill CUDA, just sign CUDA drivers on macOS instead and quit pretending like MPS is a world-class solution. There are trillions on the table, this is not an unsolvable issue.

Admittedly, my use of CUDA and Metal is fairly surface-level. But I have had great success using LLMs to convert whole gaussian splatting CUDA codebases to Metal. It's not ideal for maintainability and not 1:1, but if CUDA was a moat for NVIDIA, I believe LLMs have dealt a blow to it.

You can convert CUDA codebases to Vulkan and DirectX code, for all the good it does you. You're still constrained by the architecture of the GPU, and Apple Silicon GPUs pre-M5 are all raster-optimized. The hardware is the moat.

Apple technically hasn't supported the professional GPGPU workflow for over a decade. macOS doesn't support CUDA anymore, Apple abandoned OpenCL on all of their platforms and Metal is a bare-minimum effort equivalent to what Windows, Android and Linux get for free. Dedicated matmul hardware is what Apple should have added to the M1 instead of wasting silicon on sluggish, rinky-dink NPUs. The M5 is a day late and a dollar short.

According to reports, even Apple can't quite justify using Apple Silicon for bulk compute: https://9to5mac.com/2026/03/02/some-apple-ai-servers-are-rep...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#244

Post by bigyabai »

pdpi wrote: post #112
bigyabai wrote: post #52

Earlier quoted context omitted.

> We haven't had phones running laptop-grade CPUs/GPUs for that long Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.

Kind of. We've had solid CPUs for a while, but GPUs have lagged behind (and they're the ones that matter for this particular application). iPhones still lead by a comfortable margin on this front, but have historically been pretty limited on the IO front (only supported USB2 speeds until recently).

The GPUs are perfectly solid. Cheap Android handsets have shipped with Vulkan compliance for almost a decade now; the GPUs are equally-featured to consoles and PCs. The same goes for Apple handsets that run byte-identical Metal Compute Shaders to the Mac. For desktop use they are perfectly amenable. The hardware lacks nothing required for inference or gaming that dGPUs ordinarily support.

And even if you raise the requirements, we still have to contend with cheap CUDA-capable GPUs like the one in the ($300!!!) Nintendo Switch, or the Jetson SOCs. The mobile market has had tons of high-speed/low-power options for a very long time now.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#245

Post by aidenn0 »

HPsquared wrote: post #213
andix wrote: post #143

My iPad Air with M2 can run local LLMs rather well. But it gets ridiculously hot within seconds and starts throttling.

I wonder if anyone has made a liquid cooling system for ipads / phones. Like, a sealed thing that seals onto the back of the device and circulates cooling water directly against the back surface.

What about submerging it in mineral oil?

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#246

Post by mnkyprskbd »

bigyabai wrote: post #52
pdpi wrote: post #11

Earlier quoted context omitted.

It's both. We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.

> We haven't had phones running laptop-grade CPUs/GPUs for that long Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.

We had LLMs for about 5 minutes or so. Hardly a measure of time for an industry that goes back half a century and then some.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#247

Post by cercatrova »

HPsquared wrote: post #213
andix wrote: post #143

My iPad Air with M2 can run local LLMs rather well. But it gets ridiculously hot within seconds and starts throttling.

I wonder if anyone has made a liquid cooling system for ipads / phones. Like, a sealed thing that seals onto the back of the device and circulates cooling water directly against the back surface.

You're in luck. Lots of phone manufacturers also implement liquid cooling inside the phone too.

https://www.notebookcheck.net/Xiaomi-launches-new-mobile-wat...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#248

Post by yencabulator »

Qwen3.5-397B-A17B behaves more like a 17B parameter model. Omitting the MoE part from the headline makes it a lie and stupid hype.

Quantizing is also a cheat code that makes the numbers lie, next up someone is going to claim running a large model when they're running a 1-bit quantization of it.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#249

Post by DrewADesign »

NetMageSCW wrote: post #222
root_axis wrote: post #166

Earlier quoted context omitted.

It will never be possible on a smart phone. I know that sounds cynical, but there's basically no path to making this possible from an engineering perspective.

No one needs more than 640K!

Quantum computing is right around the corner!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#250

Post by BoorishBears »

yencabulator wrote: post #248

Qwen3.5-397B-A17B behaves more like a 17B parameter model. Omitting the MoE part from the headline makes it a lie and stupid hype. Quantizing is also a cheat code that makes the numbers lie, next up someone is going to claim running a large model when they're running a 1-bit quantization of it.

It behaves more like a ~80B parameter model (geometric mean of active and total params), and has world knowledge closer to a 400B parameter model

There's no misleading here, they show every detail from model to quantization to that atrocious time to first token. Stuff like this feels more like code golf than anyone claiming the mainstream phone user is going to even download 100GB of model weights.

Post reply on HN