Total gimmick. I guess we're "making progress", but this is will never lead to any useful application other than "Yes, you're absulotely right" bots. What's needed for real applications is 10000× the input token context and 10× the output token speed, so we're off by a factor of ... 100,000×?
iPhone 17 Pro Demonstrated Running a 400B LLM
231–240 of 362 posts
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#232[dead]
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#233Earlier quoted context omitted.
The speed on a constrained device isn't entirely the point. Two years ago, LLMs failed at answering coherently. Now... You're absolutely right. Now, LLMs are too slow to be useful on handheld devices, and the future of LLMs is brighter than ever. LLMs can be useful, but quite often the responses are about as painful as LinkedIn posts. Will they get better? Maybe. Will they get worse? Maybe.
> Will they get better? Maybe. Will they get worse? Maybe. I find it hard to understand your uncertainty; how could they not keep getting even better when we've been seeing qualitative improvements literally every second week for months on end? These improvements being eminently public and applied across multiple relevant dimensions: raw inference speed ( https://github.com/ggml-org/llama.cpp/releases ), external-fac…
There are many metrics for “better” and “worse”. It is entirely possible for an AI system to be better in the sense of hallucination while also being of less utility. An arrogant prick who’s always correct isn’t always a good person to have on your team, right?
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#234I installed Termux on an old Android phone last week (running LineageOS), and then using Termux installed Ollama and a small model. It ran terribly, but it did run.
Fwiw, my pixel 8 runs Qwen3.5 4B with 2 tok/s speed. Via pocketpal app. Somehow cactus app didn't work.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#235This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
Probably 15 to 20 years, if ever. This phone is only running this model in the technical sense of running, but not in a practical sense. Ignore the 0.4tk/s, that's nothing. What's really makes this example bullshit is the fact that there is no way the phone has a enough ram to hold any reasonable amount of context for that model. Context requirements are not insignificant, and as the context grows, the speed of the o…
KV-cache is still quite small compared to the weights. It can stay in memory for reasonable context length, or be streamed to storage as a last resort. This actually doesn't impact performance too much, since we were already limited by having to stream in the much larger weights.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#236"0.6 t/s"
This is a toy.
We need to build open infrastructure in the cloud capable of hosting a robust ecosystem of open weights.
And then we need to build very large scale open weights.
That's the only way we don't get owned by the hyperscalers.
At the edge isn't going to happen in a meaningful way to save us.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#237"0.6 t/s" This is a toy. We need to build open infrastructure in the cloud capable of hosting a robust ecosystem of open weights. And then we need to build very large scale open weights. That's the only way we don't get owned by the hyperscalers. At the edge isn't going to happen in a meaningful way to save us.
Is it though? I would say 'proof of concept' instead.
The fact that it's running on a phone now just sets the goalpost and gets everyone excited about it: add more RAM and GPU to the next iPhone and it's not a toy anymore. Co-incidentally, phone companies also have thousands of engineers sitting around wondering what to do in their next release to convince consumers to buy ...
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#238[dead]
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#239"0.6 t/s" This is a toy. We need to build open infrastructure in the cloud capable of hosting a robust ecosystem of open weights. And then we need to build very large scale open weights. That's the only way we don't get owned by the hyperscalers. At the edge isn't going to happen in a meaningful way to save us.
Is it though? I would say 'proof of concept' instead. The fact that it's running on a phone now just sets the goalpost and gets everyone excited about it: add more RAM and GPU to the next iPhone and it's not a toy anymore. Co-incidentally, phone companies also have thousands of engineers sitting around wondering what to do in their next release to convince consumers to buy ...
'Toy' and 'proof of concept' are synonymous. What this really opens up is running non-toy models like Qwen3.5 35B-A3B, which are still considered very large in the mobile device context. Yes, it's too slow for interactivity, but if you acknowledge that it's supposed to deliver "Pro" level inference it works quite fine.
Re: iPhone 17 Pro Demonstrated Running a 400B LLM
#240"0.6 t/s" This is a toy. We need to build open infrastructure in the cloud capable of hosting a robust ecosystem of open weights. And then we need to build very large scale open weights. That's the only way we don't get owned by the hyperscalers. At the edge isn't going to happen in a meaningful way to save us.
Is it though? I would say 'proof of concept' instead. The fact that it's running on a phone now just sets the goalpost and gets everyone excited about it: add more RAM and GPU to the next iPhone and it's not a toy anymore. Co-incidentally, phone companies also have thousands of engineers sitting around wondering what to do in their next release to convince consumers to buy ...
> add more RAM and GPU to the next iPhone and it's not a toy anymore
We're not going to get more RAM and GPU in consumer devices.
All of the supply is going into data center build outs. As the hyper scaler gamble on the future continues, we get left with weaker (or more expensive) devices - not stronger ones.
The market makers make more money if we're left to thin clients. They're also the ones who control supply and the shapes of devices.