Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

291–300 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#291

Post by cmiles8 »

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases.

That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion.

Apple has sat on the sidelines for much of this as it seems clear they know the end game is everyone just does this stuff locally on their phone or computer and then it’s game over for everything going on now.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#292

Post by wiseowise »

SecretDreams wrote: post #196
CrzyLngPwd wrote: post #186

I had a dream that everyone had super intelligent AIs in their pockets, and yet all they did was doomscroll and catfish...shortly before everything was destroyed.

A modern Nostradamus?

You know, Quasimodo predicted all of this.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#294

Post by alpineman »

ottah wrote: post #63
_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

Probably 15 to 20 years, if ever. This phone is only running this model in the technical sense of running, but not in a practical sense. Ignore the 0.4tk/s, that's nothing. What's really makes this example bullshit is the fact that there is no way the phone has a enough ram to hold any reasonable amount of context for that model. Context requirements are not insignificant, and as the context grows, the speed of the o…

Agree with the first part - but I can run GPT OSS 20b, a highly capable model on my laptop with 32GB of RAM at speeds that for all practical intents is as fast as GPT-5.4 and good enough for 90%+ of non-technical use cases.

As such I can't agree with "The only hope for a handheld execution of a practical, and capable AI model is both an algorithmic breakthrough" - we are much closer than 15/20 years to get these on a phone

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#295

Post by Idesmi »

DiscourseFan wrote: post #287
iLemming wrote: post #261

Earlier quoted context omitted.

The Anthropic logo is just Kurt Vonnegut’s drawing of an asshole: https://scienceleadership.org/thumbnail/34729/1920x1920 Just in case if someone still didn't realize - we do live in Idiocracy https://www.youtube.com/watch?v=gGlJgU9x8tM

I think the first thing is just a funny little literary allusion for those in the know. I mean isn’t it kind of hilarious that a company valued at $300 billion has a drawing of an asshole for its logo?

If they really wanted to honour Kurt Vonnegut, Anthropic wouldn't exist.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#296

Post by madwolf »

mlsu wrote: post #137
russellbeattie wrote: post #58

I have some macro opinions about Apple - not sure if I'm correct, but tell me what you think. Apple has always seen RAM as an economic advantage for their platform: Make the development effort to ensure that the OS and apps work well with minimal memory and save billions every year in hardware costs. In 2026, iPhones still come with 8Gb of RAM, Pro/Max come with 12Gb. The problem is that AI (ML/LLM training and infer…

Models on the phone is never going to make sense. If you're loading gigabytes of model weights into memory, you're also pushing gigabytes through the compute for inference. No matter how you slice it, no matter how dense you make the chips, that's going to cost a lot of energy. It's too energy intensive, simple as. "On device" inference (for large LLM I mean) is a total red herring. You basically never want to do it…

Living through all mobile phone history, from non-existant when I was a child to today's smartphones, I would hesitate to use such absolute phrases like "X on the phone is never going to make sense". How many things we're doing on a phone today that we wouldn't dream of 20 years ago? Local models on phones don't make sense today but in 5 years? who knows...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#297

Post by Mrngl1991 »

Aurornis wrote: post #149
zozbot234 wrote: post #148

Earlier quoted context omitted.

The writeup from the earlier experiment (running on a MacBook Pro) shows quite clearly that expert routing choices are far from uniform, and that some layer-experts are only used rarely. So you can save some RAM footprint even while swapping quite rarely.

I understand, but this isn't just a matter of not caching some experts. This is a 397B model on a device with 12GB of RAM. It's basically swapping experts out all the time, even if the distribution isn't uniform. When the individual expert sizes are similar to the entire size of the RAM on the device, that's your only option.

[dead]

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#298

Post by draxil »

cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#299

Post by zozbot234 »

alpineman wrote: post #294
ottah wrote: post #63

Earlier quoted context omitted.

Probably 15 to 20 years, if ever. This phone is only running this model in the technical sense of running, but not in a practical sense. Ignore the 0.4tk/s, that's nothing. What's really makes this example bullshit is the fact that there is no way the phone has a enough ram to hold any reasonable amount of context for that model. Context requirements are not insignificant, and as the context grows, the speed of the o…

Agree with the first part - but I can run GPT OSS 20b, a highly capable model on my laptop with 32GB of RAM at speeds that for all practical intents is as fast as GPT-5.4 and good enough for 90%+ of non-technical use cases. As such I can't agree with "The only hope for a handheld execution of a practical, and capable AI model is both an algorithmic breakthrough" - we are much closer than 15/20 years to get these on a…

With this work you can run a medium-sized model like GPT OSS 20b at native speed even while keeping those 32GB RAM almost fully available for other uses - the model seamlessly starts to slow down as RAM requirements increase elsewhere in the system and the fs cache has to evict more expert layers, and reaches full speed again as the RAM is freed up. It adds a key measure of flexibility to the existing AI local inference picture.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#300

Post by DiscourseFan »

Idesmi wrote: post #295
DiscourseFan wrote: post #287

Earlier quoted context omitted.

I think the first thing is just a funny little literary allusion for those in the know. I mean isn’t it kind of hilarious that a company valued at $300 billion has a drawing of an asshole for its logo?

If they really wanted to honour Kurt Vonnegut, Anthropic wouldn't exist.

I'm not sure what you mean by that
Post reply on HN