11–20 of 210 posts
I wonder if we'll see these models running on the phone (aiPhone) hardware in the future.
> a ∼3B-parameter on-device model
I wonder if we'll see these models running on the phone (aiPhone) hardware in the future.
It does. You can use it directly on iOS 26 beta - without writing a line of code I can toy with the on-device model through Shortcuts on my 16 Pro. It’s not meant to be a general purpose chatbot… but it can work as a general purpose chatbot in airplane mode which is a novel experience. https://share.icloud.com/photos/018AYAPEm06ALXciiJAsLGyuA https://share.icloud.com/photos/0f9IzuYQwmhLIcUIhIuDiudFw The above took li…
Earlier quoted context omitted.
It does. You can use it directly on iOS 26 beta - without writing a line of code I can toy with the on-device model through Shortcuts on my 16 Pro. It’s not meant to be a general purpose chatbot… but it can work as a general purpose chatbot in airplane mode which is a novel experience. https://share.icloud.com/photos/018AYAPEm06ALXciiJAsLGyuA https://share.icloud.com/photos/0f9IzuYQwmhLIcUIhIuDiudFw The above took li…
It would be interesting to see the tok/s comparison between the ANE and GPU for inference. I bet these small models are a lot friendlier than the 7B/12B models that technically fit on a phone but won't accelerate well without a GPU.
I wonder if we'll see these models running on the phone (aiPhone) hardware in the future.
It does. You can use it directly on iOS 26 beta - without writing a line of code I can toy with the on-device model through Shortcuts on my 16 Pro. It’s not meant to be a general purpose chatbot… but it can work as a general purpose chatbot in airplane mode which is a novel experience. https://share.icloud.com/photos/018AYAPEm06ALXciiJAsLGyuA https://share.icloud.com/photos/0f9IzuYQwmhLIcUIhIuDiudFw The above took li…
Why give this to developers if you haven’t been able to get Siri to use it yet? Does it not work or something? I guess we’ll find out when devs start trying to make stuff
Earlier quoted context omitted.
It would be interesting to see the tok/s comparison between the ANE and GPU for inference. I bet these small models are a lot friendlier than the 7B/12B models that technically fit on a phone but won't accelerate well without a GPU.
I thought the big difference between the GPU and ANE was that you couldn't use the ANE to train. Does the GPU actually perform faster during inference as well? Is that because the ANE are designed more for efficiency or is there another bigger reason?
The dozens of "contributors" being presented in random order is, one would suppose, an anti-poaching tactic?
Most of his team are former Google brain so GDM knows who is good.
Every time I see a paper from Apple I just feel like, OK so why isn’t my iPhone actually doing any of this yet? Why give this to developers if you haven’t been able to get Siri to use it yet? Does it not work or something? I guess we’ll find out when devs start trying to make stuff
Probably Apple is trying to distill the models so they can run on your phone locally. Remember, most, if not all, of Siri is running on your device. There's no round trip whatsoever for voice processing.
Also, for larger models, there will be throwaway VMs per request, so building that infra takes time.
Every time I see a paper from Apple I just feel like, OK so why isn’t my iPhone actually doing any of this yet? Why give this to developers if you haven’t been able to get Siri to use it yet? Does it not work or something? I guess we’ll find out when devs start trying to make stuff
What exactly are you referring to? Models do run on iPhone and there are features that take advantage of it, today.