Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

281–290 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#281

Post by LeanderK »

jonplackett wrote: post #275
ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

I wonder if they’d create a simple version that lives on device that can call on a ‘cleverer’ version for more extensive tasks - like how ChatGPT is using plugins. Most interactions probably don’t require full power

yeah, some sort of caching. Small models for most tasks with a way bigger models just one get-request away. It's a smart idea that hasn't been tried yet. But I bet that you could shrink the models significantly. It doesn't really have to know so much if it can google, but I we don't really know how to train such models (we just want the "common sense" without so much knowledge).

Re: Transformer architecture optimized for Apple Silicon

#282

Post by spacebanana7 »

dachryn wrote: post #278
endisneigh wrote: post #44

Earlier quoted context omitted.

yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.

dont underestimate how many of those parameters are actually necessary to support multiple languages. If you focus on english only, this can easily reduce the paramters 5fold

Could you explain how supporting multiple languages increases the parameter count so much? I'm genuinely curious.

LLMs seem to be comfortable with hundreds of programming languages, DSLs and application specific syntaxes so how does supporting a couple more natural languages become so expensive?

I see how more training data would be needed, but I don't understand how that maps to a greater parameter count.

Re: Transformer architecture optimized for Apple Silicon

#283

Post by Cthulhu_ »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Definitely, for some years now the iPhone has had AI stuff on board to categorize and organize your photos (recognising people, creatures, "things" etc), all on board and built into the device - unlike Google that requires you to upload them first. It's not a stretch at all to have more functionality like Siri run on-device.

Re: Transformer architecture optimized for Apple Silicon

#284

Post by machinekob »

callen43 wrote: post #271

Is there also something available to make use of the ANE during _training_? E.g fine-tuning BERT on an M1 Mac in a couple of hours? (This here only applies to inference, right?)

Its FP16/Int8 inference only (cause you can only access it via apple frameworks that dosent support training). Also its only used if your data is small enough (4mb cache) it wont be useful for big transformers/ big images processing in a while.

Re: Transformer architecture optimized for Apple Silicon

#285

Post by blueboo »

qgin wrote: post #104

In regards to LLMs there is a collision between Apple's extremely good chip design capabilities and Apple's insistence that Siri never says anything that isn't 100% scripted and 100% certain to not bad. Up until now, they've chosen to limit Siri functionality rather than leave anything to chance. LLMs will absolutely be able to run locally, but whether Apple will be able to stop worrying and love the model remains to…

they're waiting for the overton window to shift wrt acceptance of imperfect LLM responses...

...it's shifting...

Re: Transformer architecture optimized for Apple Silicon

#286

Post by christkv »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

One step at a time towards the "family AI's" of Neuromancer.

Re: Transformer architecture optimized for Apple Silicon

#287

Post by yyyk »

davnicwil wrote: post #19
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

> Apple has a ridiculous, almost unfathomably deep moat for.. running personalised...

Apple has decent hw but no LLM software. Unfortunately, in LLM space software changes are the ones driving performance for now, since the space hasn't stabilized yet. Since their competitors control the software, they get to adapt it to their hardware. That is, Google and Microsoft are going to adapt GPT/Bard to Qualcomm ARM etc. while Apple is being ignored. Unless Apple gets in with their own LLM (quite possible), their hw advantage will end up not mattering one bit.

Re: Transformer architecture optimized for Apple Silicon

#288

Post by hardware2win »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

OpenAI and MSFT couldnt figure it out and are burning milions of USD on compute, but Apple will make it run on phone in 5 years?

Crazy bet, what makes you think that? You cannot optimize infinitely. Raytracing probably had decades of ppl trying to make it run fast and yet even today you need strong hardware

Re: Transformer architecture optimized for Apple Silicon

#289

Post by niklasrde »

moonchrome wrote: post #261
post-it wrote: post #221

Earlier quoted context omitted.

It's probably not glasses. Nobody likes wearing glasses. Every attempt at making tech glasses has failed. It'll probably be the Apple Watch.

I'm not a VR/AR fan but if Apple can come out with super high quality AR headset that can augment/replace monitors I would buy one instantly. I might be earlier than mainstream adopter but if the tech ever gets there this will be a natural shift. I won't buy Meta products but I keep waiting for someone to do VR workspace right - where I can truly work and travel from anywhere with a decent chair and a desk for keyboa…

I think that's it. I had half an hour to kill in a cafe yesterday, didn't bring my kindle, and I thought "if there were compact AR or even VR glasses that I could whip out to emulate a full-size working environment, that would be super handy right now".

Having said that, for me to spend money on it it would (a) also be able to realistically replace my set up at home, which currently consists of 2 HiDPI screens and (b) not be that much more expensive than what that cost me.

Re: Transformer architecture optimized for Apple Silicon

#290

Post by datpiff »

pornel wrote: post #120
s1k3s wrote: post #39

Earlier quoted context omitted.

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

Hundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds. It's also likely that memory requirements will be shrunk through software and ML improvements.

> Hundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds.

It's not expected to. The consensus seems to be ~2025 https://arxiv.org/abs/1511.05956

Post reply on HN