Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

181–190 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#181

Post by happyopossum »

Aurornis wrote: post #163
alwillis wrote: post #159

Earlier quoted context omitted.

> Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design. > Mobile phones don't have separate GPUs and separate VRAM like some desktops. That's true. The difference is the iPhone has wider memory buses and uses faster LPDDR5 memory. Apple places the RAM dies directly on the same package as the SoC (PoP — Package on Package), minimizing latency. Some Android…

> The difference is the iPhone has wider memory buses and uses faster LPDDR5 memory. Apple places the RAM dies directly on the same package as the SoC (PoP — Package on Package), minimizing latency. Some Android phones have started to do this, too. Package-on-Package has been used in mobile SoCs for a long time. This wasn't an Apple invention. It's not new, either. It's been this way for 10+ years. Even cheap Raspber…

> The memory bandwidth of flagship iPhone models is similar to the memory bandwidth of flagship Android phones

More correct to say that the memory bandwidth of ALL iPhone models is similar to the memory bandwidth of flagship Android models. The A18 and A18 pro do not differ in memory bandwidth.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#182

Post by alwillis »

ecshafer wrote: post #101
russellbeattie wrote: post #58

I have some macro opinions about Apple - not sure if I'm correct, but tell me what you think. Apple has always seen RAM as an economic advantage for their platform: Make the development effort to ensure that the OS and apps work well with minimal memory and save billions every year in hardware costs. In 2026, iPhones still come with 8Gb of RAM, Pro/Max come with 12Gb. The problem is that AI (ML/LLM training and infer…

In a recent episode of Dwarkesh the guest who is a semiconductor industry analyst predicted that an iPhone will increase in price by about $250 for the same stuff due to increased ram/chip costs from AI. Apple will not be able to afford to put a bunch more RAM into the phones and still sell them.

> In a recent episode of Dwarkesh the guest who is a semiconductor industry analyst predicted that an iPhone will increase in price by about $250 for the same stuff due to increased ram/chip costs from AI. Apple will not be able to afford to put a bunch more RAM into the phones and still sell them.

Apple recently stated on an earnings call they signed contracts with RAM vendors before prices got out of control, so they should be good for a while. Nvidia also uses TSMC for their chips, which may affect A series and M series chip production.

Yes, TSMC has a plant in Arizona but my understanding is they can't make the cutting edge chips there; at least not yet.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#183

Post by jgraham »

johnwhitman wrote: post #154

[flagged]

Power in general.

Your time-average power budget for things that run on phones is about 0.5W (batteries are about 10Wh and should last at least a day). That's about three orders of magnitude lower than a the GPUs running in datacenters.

Even if battery technology improves you can't have a phone running hot, so there are strong physical limits on the total power budget.

More or less the same applies to laptops, although there you get maybe an additional order of magnitude.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#184

Post by mordechai9000 »

noboostforyou wrote: post #175
MasterScrat wrote: post #171

Earlier quoted context omitted.

This has a simple pragmatic solution though: https://duckdb.org/2024/12/06/duckdb-tpch-sf100-on-mobile#a-...

From the same article: "The phone a few minutes after finishing the benchmark. It no longer booted because the battery was too cold!"

Removing the case and putting it in mineral oil with a circulating pump and a heat exchanger would probably work better

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#185

Post by geek_at »

root_axis wrote: post #164
yalogin wrote: post #132

Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway. I understand this is for a demo but do we really need a 400B model in the mobile? A 10B model would do fine right? What do we miss with a pared down one?

Compared to a 400b model, a 10b is practically useless, it's not even worth bothering outside of tinkering for fun and research.

Still dreaming about an android keyboard that plugs into local or self hosted llm backend for smarter text predictions

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#187

Post by ankaz »

Tade0 wrote: post #34
_air wrote: post #23

This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains

Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html

[dead]
Post reply on HN