Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

iPhone 17 Pro Demonstrated Running a 400B LLM

twitter.com

Post reply on HN
Print view View original

341–350 of 362 posts

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#341

Post by sabon »

latexr wrote: post #340
dudefeliciano wrote: post #333

Earlier quoted context omitted.

"PS. This post is meant to be humorous, but let's not pretend there isn't a serious point here about the depressing sameness in modern design. No actual anuses were consulted during this research, though several designers were clearly thinking about them." > Was that even written by a human? https://velvetshark.com/

> This post is meant to be humorous Was anyone supposed to think a post about comparing logos to buttholes was meant to be serious? Either way, the joke doesn’t work if what you’re describing makes no sense (circle and gradient) and are stretching the definition to unrecognizability. > > Was that even written by a human? > https://velvetshark.com/ So, probably not: > I build AI agent systems and help companies implem…

It was written by a human.

Source: I'm a human. I wrote it.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#342

Post by efsavage »

cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

> “good enough” for 95% of use cases

Maybe, for current use cases. I'd argue that anyone who thinks they can do everything a 10kW server can do on their 10W device just isn't being creative enough :)

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#344

Post by hiddencost »

cmiles8 wrote: post #291

To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…

Consumer market is small compared to headcount reduction and cutting edge science.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#345

Post by rogerrogerr »

classified wrote: post #330
bkfh wrote: post #315

Earlier quoted context omitted.

Thanks, bot.

Wouldn't a bot write better English? Or are they optimized to produce bad grammar already?

This isn't bad grammar, it's bad formatting because it was copy-pasted from somewhere and the newlines didn't take.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#346

Post by efnx »

plussed_reader wrote: post #331
noemit wrote: post #317

Earlier quoted context omitted.

Even if it runs, this will run slowly, and heat up. I think local will always have a place, but the infrastructure is going to be used in my humble opinion.

I don't want to put information into a black box of mystery that can then be used for other monetization purposes. I am still waiting for a realistic local solution.

Have you tried qwen3.5 running locally? It’s quite “good enough”.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#347

Post by stingraycharles »

zozbot234 wrote: post #319
fouc wrote: post #312

Earlier quoted context omitted.

looks like they meant “effective dense size” which is the square root of total params×active params, so in this case sqrt(397 x 17) = ~82

But the claim that "one expert is 17B" is incorrect. Experts are picked with per-layer granularity (expert 1 for layer X may well be entirely unrelated to expert 1 for layer Y), and the individual layer-experts are tiny. The writeup for the original experiment is very clear on this.

Ok I am by no means an expert on this and I immediately stand corrected. But as I understand it, in order to understand the amount of active memory that’s required, it’s more accurate to go by the ~82B number, right?

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#348

Post by harlanji »

Yizahi wrote: post #325
draxil wrote: post #298

Earlier quoted context omitted.

I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.

We really can't have open source LLM, because they are all based on the stolen IP, or stolen IP slightly laundered and under different title.

I feel like an opt-in model built on AGPL code should output AGPL code.

I'd put my work into that. Not the only option just an example.

Every great project takes time to build. It's possible.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#349

Post by mlsu »

madwolf wrote: post #296
mlsu wrote: post #137

Earlier quoted context omitted.

Models on the phone is never going to make sense. If you're loading gigabytes of model weights into memory, you're also pushing gigabytes through the compute for inference. No matter how you slice it, no matter how dense you make the chips, that's going to cost a lot of energy. It's too energy intensive, simple as. "On device" inference (for large LLM I mean) is a total red herring. You basically never want to do it…

Living through all mobile phone history, from non-existant when I was a child to today's smartphones, I would hesitate to use such absolute phrases like "X on the phone is never going to make sense". How many things we're doing on a phone today that we wouldn't dream of 20 years ago? Local models on phones don't make sense today but in 5 years? who knows...

Because for every increase in efficiency that you get on the phone, you get on the datacenter too. (and likely on the modem as well).

The gap will always be there. If the silicon gets efficient enough to compute a question/response on the phone in 1 joule, the datacenter will be able to do it with a way smarter way better model in 0.1 joule. And also if the silicon gets efficient enough, that means everything else on the phone will get more efficient too and the battery will get smaller and lighter, so 1 joule will be more 'expensive' relative to the battery SOC. It will never make sense no matter how good the silicon gets.

We have GPT-4 level performance in 22b models today. Only a tiny tiny minority actually use those, because opus is that much better. When it comes to energy efficiency the bar gets higher everywhere in inference and training.

Re: iPhone 17 Pro Demonstrated Running a 400B LLM

#350

Post by zozbot234 »

stingraycharles wrote: post #347
zozbot234 wrote: post #319

Earlier quoted context omitted.

But the claim that "one expert is 17B" is incorrect. Experts are picked with per-layer granularity (expert 1 for layer X may well be entirely unrelated to expert 1 for layer Y), and the individual layer-experts are tiny. The writeup for the original experiment is very clear on this.

Ok I am by no means an expert on this and I immediately stand corrected. But as I understand it, in order to understand the amount of active memory that’s required, it’s more accurate to go by the ~82B number, right?

The ~82B figure is an attempt to compare performance to an equivalent dense model. The amount of active parameters is given by the ~17B.
Post reply on HN