351–360 of 362 posts
My iPad Air with M2 can run local LLMs rather well. But it gets ridiculously hot within seconds and starts throttling.
I wonder if anyone has made a liquid cooling system for ipads / phones. Like, a sealed thing that seals onto the back of the device and circulates cooling water directly against the back surface.
It's a nice experiment, but I really wonder what's the use case? Privacy, yes. Local, yes. But then? Will people really use an LLM in their iPhone while they can use LLM infrastructure with bigger models for complex tasks? I mean, it really looks cool. But I don't think it's gonna be the future of local AI also. Maybe someone who can build up a very specialized local model for one particular task can enjoy that. Not…
I expect OpenAI, Anthropic, and other companies will attempt to do the same, but the OS manufacturers will have a step up.
To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…
Even if it runs, this will run slowly, and heat up. I think local will always have a place, but the infrastructure is going to be used in my humble opinion.
Earlier quoted context omitted.
> This post is meant to be humorous Was anyone supposed to think a post about comparing logos to buttholes was meant to be serious? Either way, the joke doesn’t work if what you’re describing makes no sense (circle and gradient) and are stretching the definition to unrecognizability. > > Was that even written by a human? > https://velvetshark.com/ So, probably not: > I build AI agent systems and help companies implem…
It was written by a human. Source: I'm a human. I wrote it.
Earlier quoted context omitted.
It behaves more like a ~80B parameter model (geometric mean of active and total params), and has world knowledge closer to a 400B parameter model There's no misleading here, they show every detail from model to quantization to that atrocious time to first token. Stuff like this feels more like code golf than anyone claiming the mainstream phone user is going to even download 100GB of model weights.
I think we're using different meaning of "behaves like". I meant "has tokens/sec performance comparable to".
Earlier quoted context omitted.
Still need massive amounts of compute for training. Nobody is going to be training 400B models on a phone any time soon.
Likely not. We’re seeing a massive slowing in the value of all that additional training. Folks don’t like to talk about that, but absent a completely new break-thru the current math of LLMs has largely run its course. We simply don’t need massive training forever and ever. We’re getting to the point that “good enough” models will solve most use cases. The demonstrated business value is also still broadly missing for…
I'm adapting all of this to Rust+WGPU with compute shaders if you want to follow along.
See this repo: https://github.com/tmzt/shady-thinker
Goal is Qwen3.5 27b on a Pixel 10 Pro running GrapheneOS.
To the extent that the present LLM movement reaches a steady state conclusion it’s highly likely to be open source models on your own hardware that are “good enough” for 95% of use cases. That blows up the whole “industrial complex” being developed around massive data centers, proprietary models, and everything that goes with that. Complete implosion. Apple has sat on the sidelines for much of this as it seems clear…
I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.