Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

301–310 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#301

Post by cmiles8 »

draxil wrote: post #298
cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.

Fair clarification, yes.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#302

Post by EruditeCoder108 »

This is less about “running a 400B model on a phone” and more about clever engineering around constraints. What’s actually happening is: in mixture-of-experts only a small subset of weights is active per token Aggressive quantization Streaming weights from storage instead of loading everything into RAM So the effective working set is much smaller than 400B. That said, the trade-offs are obvious: very low token throughput, high latency, and heavy reliance on storage bandwidth. It’s more of a proof-of-concept than something usable.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#303

Post by TeMPOraL »

Tade0 wrote: post #130
RALaBarge wrote: post #116

Earlier quoted context omitted.

I think the future is the model becoming lighter not the hardware becoming heavier

The hardware will become heavier regardless I'm afraid.

Good. It's ridiculously tiny and lightweight these days.

Especially with phones; the first thing everyone does after buying their new uber thin iPhone is buying a case for it, which doubles its thickness.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#306

Post by Tade0 »

jml7c5 wrote: post #218
HPsquared wrote: post #213

Earlier quoted context omitted.

I wonder if anyone has made a liquid cooling system for ipads / phones. Like, a sealed thing that seals onto the back of the device and circulates cooling water directly against the back surface.

A more whimsical method is to put the thing in a glass of water with the cord sticking out. :-) https://www.reddit.com/r/EmulationOnAndroid/comments/1m269k0...

As I discovered cooling down hardboiled eggs, it's better to keep a thin layer of moisture that cools the object via evaporation.

Something of this sort should keep the device moisturised:

https://www.thehydrobros.com/products/automatic-water-spraye...

0.2ml/s at its lowest setting looks like the ballpark of what's required to maintain temperature.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#307

Post by dudefeliciano »

iLemming wrote: post #261
CrzyLngPwd wrote: post #186

I had a dream that everyone had super intelligent AIs in their pockets, and yet all they did was doomscroll and catfish...shortly before everything was destroyed.

The Anthropic logo is just Kurt Vonnegut’s drawing of an asshole: https://scienceleadership.org/thumbnail/34729/1920x1920 Just in case if someone still didn't realize - we do live in Idiocracy https://www.youtube.com/watch?v=gGlJgU9x8tM

there is a case to be made that all AI company logos are just drawings of assholes: https://velvetshark.com/ai-company-logos-that-look-like-butt...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#308

Post by huflungdung »

EruditeCoder108 wrote: post #302

This is less about “running a 400B model on a phone” and more about clever engineering around constraints. What’s actually happening is: in mixture-of-experts only a small subset of weights is active per token Aggressive quantization Streaming weights from storage instead of loading everything into RAM So the effective working set is much smaller than 400B. That said, the trade-offs are obvious: very low token throug…

[dead]

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#309

Post by mr_toad »

cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

Still need massive amounts of compute for training. Nobody is going to be training 400B models on a phone any time soon.
Post reply on HN