Earlier quoted context omitted.
Maybe you should have asked a better question. :P
What do you get if you multiply six by nine?
54?
iPhone 17 Pro Demonstrated Running a 400B LLM
61–70 of 362 posts
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#62This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html
I think you're ignoring the inevitable march of progress. Phones will get big enough to hold it soon.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#63This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
Probably 15 to 20 years, if ever. This phone is only running this model in the technical sense of running, but not in a practical sense. Ignore the 0.4tk/s, that's nothing. What's really makes this example bullshit is the fact that there is no way the phone has a enough ram to hold any reasonable amount of context for that model. Context requirements are not insignificant, and as the context grows, the speed of the output will be even slower.
Realistically you need +300GB/s fast access memory to the accelerator, with enough memory to fully hold at least greater than 4bit quants. That's at least 380GB of memory. You can gimmick a demo like this with an ssd, but the ssd is just not fast enough to meet the minim specs for anything more than showing off a neat trick on twitter.
The only hope for a handheld execution of a practical, and capable AI model is both an algorithmic breakthrough that does way more with less, and custom silicon designed for running that type of model. The transformer architecture is neat, but it's just not up for that task, and I doubt anyone's really going to want to build silicon for it.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#64Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."
I mean size says nothing, you could do it on a Pi Zero with sufficient storage attached. So this post is like saying that yes an iPhone is Turing complete. Or at least not locked down so far that you're unable to do it.
You need fast storage to make it worthwhile. PCIe x4 5.0 is a reasonable minimum. Or multiple PCIe x4 4.0 accessed in parallel, but this is challenging since the individual expert-layers are usually small. Intel Optane drives are worth experimenting with for the latter (they are stuck on PCIe 4.0) purely for their good random-read properties (quite aside from their wearout resistance, which opens up use for KV-cache and even activations).
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#65This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
Only way to have hardware reach this sort of efficiency is to embed the model in hardware. This exists[0], but the chip in question is physically large and won't fit on a phone. [0] https://www.anuragk.com/blog/posts/Taalas.html
That's actually pretty cool, but I'd hate to freeze a models weights into silicon without having an incredibly specific and broad usecase.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#66Earlier quoted context omitted.
> maybe even learned prefetching for what the next experts will be Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.
Manually no. It would have to be learned, and making the expert selection predictable would need to be a training metric to minimize.
Making the expert selection more predictable also means making it less effective. There's no real free lunch.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#67Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."
I too thought you were joking
laughed when it slowly began to type that out
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#68A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.
I mean, by any reasonable standard it still is. Almost any computer can run an llm, it's just a matter of how fast, and 0.4k/s (peak before first token) is not really considered running. It's a demo, but practically speaking entirely useless.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#69Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observation, and you are absolutely right ..."
Better than waiting 7.5 million years to have a tell you the answer is 42.
Should have used a better platform. So long and thanks for all the fish.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#70Earlier quoted context omitted.
Plus all those pricey 512GB Mac Studios they are selling to YouTubers.
They don't offer the 512 gig RAM variant anymore. Outside of social media influencers and the occasional AI researcher, the market for $10K desktops is vanishingly small.
The best desktop you could get has been around $10k going back all the way back to the PDP-8e (it could fit on most desks!).