Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

311–320 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#311

Post by mike_hearn »

draxil wrote: post #298
cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.

Nemotron is genuinely open source at least at the smaller sizes. You can download the datasets.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#312

Post by fouc »

zozbot234 wrote: post #285
stingraycharles wrote: post #281

Earlier quoted context omitted.

One expert is 17B, but more than one expert can be active at any time. I believe it’s actually more like 80B active.

I don't think this is correct, "active parameters" is quite unambiguous in that it means a sum of all active experts plus shared parameters.

looks like they meant “effective dense size” which is the square root of total params×active params, so in this case sqrt(397 x 17) = ~82

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#313

Post by cmiles8 »

mr_toad wrote: post #309
cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

Still need massive amounts of compute for training. Nobody is going to be training 400B models on a phone any time soon.

Likely not.

We’re seeing a massive slowing in the value of all that additional training. Folks don’t like to talk about that, but absent a completely new break-thru the current math of LLMs has largely run its course.

We simply don’t need massive training forever and ever. We’re getting to the point that “good enough” models will solve most use cases. The demonstrated business value is also still broadly missing for AI on the level required to keep funding all this training for much longer.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#314

Post by adam_patarino »

EruditeCoder108 wrote: post #302

This is less about “running a 400B model on a phone” and more about clever engineering around constraints. What’s actually happening is: in mixture-of-experts only a small subset of weights is active per token Aggressive quantization Streaming weights from storage instead of loading everything into RAM So the effective working set is much smaller than 400B. That said, the trade-offs are obvious: very low token throug…

I’ve seen this story making the rounds and I’m not just why it’s gotten so much traction. Is it just a good write up?

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#315

Post by bkfh »

EruditeCoder108 wrote: post #302

This is less about “running a 400B model on a phone” and more about clever engineering around constraints. What’s actually happening is: in mixture-of-experts only a small subset of weights is active per token Aggressive quantization Streaming weights from storage instead of loading everything into RAM So the effective working set is much smaller than 400B. That said, the trade-offs are obvious: very low token throug…

Thanks, bot.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#316

Post by alnah »

It's a nice experiment, but I really wonder what's the use case? Privacy, yes. Local, yes. But then? Will people really use an LLM in their iPhone while they can use LLM infrastructure with bigger models for complex tasks? I mean, it really looks cool. But I don't think it's gonna be the future of local AI also. Maybe someone who can build up a very specialized local model for one particular task can enjoy that. Not sure it's gonna be massively use by the common of the mortals... But fore sure, for the industry, there is maybe a direction where we could have different very specialized models, on our devices, that could interoperate together, and then, provide something useful. We'll see. Interesting though! Maybe we still need some years, or decades, before we have devices, laptops, good enough to run good models.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#317

Post by noemit »

cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

Even if it runs, this will run slowly, and heat up.

I think local will always have a place, but the infrastructure is going to be used in my humble opinion.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#318

Post by cmiles8 »

noemit wrote: post #317
cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

Even if it runs, this will run slowly, and heat up. I think local will always have a place, but the infrastructure is going to be used in my humble opinion.

Today yes, but between the improved performance of smaller on device models and the hardware itself getting better this issue is short lived.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#319

Post by zozbot234 »

fouc wrote: post #312
zozbot234 wrote: post #285

Earlier quoted context omitted.

I don't think this is correct, "active parameters" is quite unambiguous in that it means a sum of all active experts plus shared parameters.

looks like they meant “effective dense size” which is the square root of total params×active params, so in this case sqrt(397 x 17) = ~82

But the claim that "one expert is 17B" is incorrect. Experts are picked with per-layer granularity (expert 1 for layer X may well be entirely unrelated to expert 1 for layer Y), and the individual layer-experts are tiny. The writeup for the original experiment is very clear on this.
Post reply on HN