Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

91–100 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#91

Post by Terretta »

zozbot234 wrote: post #53
Aurornis wrote: post #38

Earlier quoted context omitted.

It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5. It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.

If the bottleneck is storage bandwidth that's not "slow". It's only slow if you insist on interactive speeds, but the point of this is that you can run cheap inference in bulk on very low-end hardware.

> very low-end hardware

iPhone 17 Pro outperforms AMD’s Ryzen 9 9950X per https://www.igorslab.de/en/iphone-17-pro-a19-pro-chip-uebert...

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#92

Post by alephnerd »

dv_dt wrote: post #74

CPU, memory, storage, time tradeoffs rediscovered by AI model developers. There is something new here, add GPU to the trade space.

It's been known to people working in the space for a long time. Heck, I was working on similar stuff for the Maxwell and later Pascal over a decade ago.

You do have a lot of "MLEs" and "Data Scientists" who only know basic PyTorch and SKLearn, but that kind of fat is being trimmed industry wide now.

Domain experience remains gold, especially in a market like today's.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#93

Post by svachalek »

_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

A long time. But check out Apollo from Liquid AI, the LFM2 models run pretty fast on a phone and are surprisingly capable. Not as a knowledge database but to help process search results, solve math problems, stuff like that.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#94

Post by bartread »

intrasight wrote: post #28
causal wrote: post #15

Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."

Better than waiting 7.5 million years to have a tell you the answer is 42.

Looked at a certain way it's incredible that a 40-odd year old comedy sci-fi series is so accurate about the expected quality of (at least some) AI output.

Which makes it even funnier.

It makes me a little sad that Douglas Adams didn't live to see it.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#95

Post by lesam »

ep103 wrote: post #89
intrasight wrote: post #28

Earlier quoted context omitted.

Better than waiting 7.5 million years to have a tell you the answer is 42.

Some one should let Douglas Adams know the calculation could have been so much faster if the machine just lied.

I think Adams was prescient, since in his story the all powerful computer reaches the answer '42' via incorrect arithmetic.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#96

Post by winwang »

otikik wrote: post #71
WarmWash wrote: post #26

Earlier quoted context omitted.

I don't think we are ever going to win this. The general population loves being glazed way too much.

The other day, I got: "You are absolutely right to be confused" That was the closest AI has been to calling me "dumb meatbag".

It would be much worse if it had said "You are absolutely wrong to be confused", haha.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#97

Post by breggles »

SV_BubbleTime wrote: post #85
cogman10 wrote: post #6

Earlier quoted context omitted.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

>triumph It’s been a lot of years, but all I can hear after reading that is … I’m making a note here, huge success

It's hard to overstate my satisfaction!

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#98

Post by GorbachevyChase »

SV_BubbleTime wrote: post #85
cogman10 wrote: post #6

Earlier quoted context omitted.

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

>triumph It’s been a lot of years, but all I can hear after reading that is … I’m making a note here, huge success

There’s no use crying over every mistake. You just keep on trying until you run out of cake.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#99

Post by alephnerd »

ottah wrote: post #68
ashwinnair99 wrote: post #3

A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.

I mean, by any reasonable standard it still is. Almost any computer can run an llm, it's just a matter of how fast, and 0.4k/s (peak before first token) is not really considered running. It's a demo, but practically speaking entirely useless.

Devils advocate - this actually shows how promising TinyML and EdgeML capabilities are. SoCs comparable to the A19 Pro are highly likely to be commodified in the next 3-5 years in the same manner that SoCs comparable to the A13 already are.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#100

Post by simopa »

vessenes wrote: post #87
volemo wrote: post #47

Earlier quoted context omitted.

> moving forward, as the information density and architectural efficiency of smaller models continue to increase If they continue to increase.

They will. Either new architectures will come out that give us greater efficiency, or we will hit a point where the main thing we can do is shove more training time onto these weights to get more per byte. Similar thing is already happening organically when it comes to efficient token use; see for instance https://github.com/qlabs-eng/slowrun .

Thanks for the link.
Post reply on HN