Is there also something available to make use of the ANE during _training_? E.g fine-tuning BERT on an M1 Mac in a couple of hours?
(This here only applies to inference, right?)
Transformer architecture optimized for Apple Silicon
271–280 of 342 posts
Re: Transformer architecture optimized for Apple Silicon
#272Earlier quoted context omitted.
Homepods have U1 chips. They can position airtags within your house. You can put an airtag under, or adjacent to, a lamp and link it to a smart switch. Your apple watch has gesture detection (still in beta, tbf), as well as U1 and real time positioning capabilities. Everything required for apple to know not only where you are, but which way your hand is pointing, as well as where "smart devices" are in your house is…
$25 for an Airtag, plus say $20 for a smart switch. Plus $200 for a HomePod. To turn a single lamp on/off by pointing at it, something that nobody ever wants to do (alright, once for the cool factor). If someone wanted to overpay for useless features, they can already go for a Philips Hue. I can turn off my lamps from anywhere in the world using a $10 Tuta ZigBee bridge and a $8 LIDL light.
You sir, are not Apple’s target market.
Re: Transformer architecture optimized for Apple Silicon
#273Earlier quoted context omitted.
> Now they're cool and everywhere. One of these things is true.
There is a difference between the subjective and objective "cool." They've sold something like 150 million pairs of Airpods? I think that qualifies as objectively "cool."
Another heuristic is: are the majority of attractive and young people using/buying/doing it?
AirPods are cool by that measure as well.
Re: Transformer architecture optimized for Apple Silicon
#274In regards to LLMs there is a collision between Apple's extremely good chip design capabilities and Apple's insistence that Siri never says anything that isn't 100% scripted and 100% certain to not bad. Up until now, they've chosen to limit Siri functionality rather than leave anything to chance. LLMs will absolutely be able to run locally, but whether Apple will be able to stop worrying and love the model remains to…
It would seem to be more risky if the model is out on the device and you can’t correct any screw ups without doing an update.
They’ll have to do something about Siri soon though. Even my 5 year old daughter told me Siri is ‘a bit thick’. And that’s just compared to Alexa never I’d ChatGPT level
Re: Transformer architecture optimized for Apple Silicon
#275The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.
I wonder if they’d create a simple version that lives on device that can call on a ‘cleverer’ version for more extensive tasks - like how ChatGPT is using plugins.
Most interactions probably don’t require full power
Re: Transformer architecture optimized for Apple Silicon
#276Earlier quoted context omitted.
The part that Siri is bad at will be commoditized — someone will open-source a GPT-4-level language model. And Apple's moat will be being able to run that on-device with all the attendant benefits (privacy, zero marginal cost to the company, availability in more scenarios, etc)
You realize that these language models are like 100's of GBs in size and consumes 10's of GB's of memory. Last time I checked, apple still ships their products with less than the market average in both of these specs. If you want a local running LLM on an iphone, get ready to sell a kidney.
You can today run an LLM vastly better than Siri on a few GB of RAM using Llama 7B at 4-bit quantization and alpaca.cpp. This is moving so fast, every day there is something new coming. There won't be any moat in LLMs soon or even in dedicated HW as it turns out you don't need that much for "basic intelligence".
Note I'm not suggesting you can pack the full knowledgebase of humanity into those 2GB of RAM, but the key feature of an edge AI is simply to understand instructions, something Siri and Ok Google struggle with at best..
Re: Transformer architecture optimized for Apple Silicon
#277The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.
Hyper casual games are around 300Mb these days, proper AAA games are multiple GB. People still download those, as you can tell by the billions of dollars they make. The problem with OpenAI's business model is that it's actually quite expensive for them to maintain centralised processing. With Apple, there are billions of very powerful computers deployed to users and these computers mostly stay idle apart from occasio…
It might actually be _good_ for apple to have a genuine reason to get a new phone.
There really isn’t that much difference between iPhone 12 and 14
If a nee one comes out with LLM Siri + hardware that makes it possible that would be a massive upgrade cycle.
Re: Transformer architecture optimized for Apple Silicon
#278Earlier quoted context omitted.
Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?
yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.
dont underestimate how many of those parameters are actually necessary to support multiple languages.
If you focus on english only, this can easily reduce the paramters 5fold
Re: Transformer architecture optimized for Apple Silicon
#279Earlier quoted context omitted.
Alpaca already works in just 4GB of RAM. This stuff is moving incredibly fast.
It doesn't work well at all. (Also alpaca.cpp uses 8GB of RAM for a 4GB model, possibly because they don't know how mmap works.)
alpaca lora is quite capable actually and is suddenly gaining a hell of a lot of traction in the corporate world
Re: Transformer architecture optimized for Apple Silicon
#280Earlier quoted context omitted.
Just tried this exact interaction. Siri gave me a list of nearest Toyotas and made the call. Seems to be working just fine. Starting to think all these Siri complaints are either made up or really outdated.
You must be the one user that Siri works perfectly for.
I spent a decade lowering my expectations, and they're still not low enough.