Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

51–60 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#51

Post by guestbest »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Apple will never have their tech run locally for AI anymore than they allow Siri to search the internet offline or while driving. Apple will only build the client as an html wrapping or inside an app but it will always run on apple servers

Re: Transformer architecture optimized for Apple Silicon

#52

Post by mlboss »

Gigachad wrote: post #27
ll_mama wrote: post #18

Earlier quoted context omitted.

5 years? Bit of a long stretch with how much focus is going into AI. I’d be surprised if it’s not possible within two iterations of the new iPhone.

GPT-3 was said to require something like 150gb of VRAM. I don't see that gap being bridged in phones within 2 years.

LLaMA already runs on M2 and is comparable to gpt 3

Re: Transformer architecture optimized for Apple Silicon

#53

Post by PaulDavisThe1st »

Gigachad wrote: post #27
ll_mama wrote: post #18

Earlier quoted context omitted.

5 years? Bit of a long stretch with how much focus is going into AI. I’d be surprised if it’s not possible within two iterations of the new iPhone.

GPT-3 was said to require something like 150gb of VRAM. I don't see that gap being bridged in phones within 2 years.

Not impossible that the gap will be bridged in the other direction, by GPT-N or a cousin requiring much less VRAM ....

Re: Transformer architecture optimized for Apple Silicon

#54

Post by bradDonniger »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

They under estimate chip makers plans to embed AI in silicon.

Chips intended for launch in 5-6 years are in planning stages right now. Apple, nVidia, and Intel could bring serious hurt to software companies in the next 5-10.

Open source purists will weep but really most people do not care, and tech should not merely serve the dedicated.

Re: Transformer architecture optimized for Apple Silicon

#55

Post by endisneigh »

brundolf wrote: post #49
endisneigh wrote: post #36

Earlier quoted context omitted.

apple fitness I imagine has a pretty small opex (sure they pay trainers, but that doesn't scale linearly with subscriptions. the videos themselves are limited in scope and could be cached pretty trivially. the actual fitness tracking happens on device), same with apple arcade which I highly doubt has any additional expensive over the app store in general. I can see it - "Siri+", pay $5 a month for fine tuned model up…

You don't need a subscription to track and view your data (I've got an apple watch and no subscription). I think all the subscription gets you are the training courses (videos, but also live events and stuff)

my point was more around how you need apple fitness + subscription to do relatively trivial things like show your health metrics on a TV (which you also need an apple TV for).

Re: Transformer architecture optimized for Apple Silicon

#56

Post by month13 »

s1k3s wrote: post #47
endisneigh wrote: post #44

Earlier quoted context omitted.

yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.

Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?

This is an older paper, but DeepMind alleges in their Chinchilla paper that far better performance can be extracted with fewer parameters; quote

"We find that current large language models are significantly under-trained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."

It's difficult to evaluate a LLM's performance as it's all qualitative, but Meta's LLaMA has been doing quite well, at even 13B parameters.

Re: Transformer architecture optimized for Apple Silicon

#57

Post by Pulcinella »

guestbest wrote: post #51
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Apple will never have their tech run locally for AI anymore than they allow Siri to search the internet offline or while driving. Apple will only build the client as an html wrapping or inside an app but it will always run on apple servers

Certain Siri features (and other AI features like photo subject identification) already run locally.

Re: Transformer architecture optimized for Apple Silicon

#58

Post by l33tman »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

You can already run stuff like it on mobiles, for example alpaca.cpp with Llama models run fine on the CPU on any device that has a few gigs of RAM. Getting to even larger models using weight quantization, distilling etc it's not 5 years, it's 5 weeks or maybe 5 months at most..

Btw transformers are really simple and the optimisations making them run fast on CPUs have come a long way. I don't know the M1/M2 benchmarks for this, but many CPUs for edge devices have NN accelerators on the silicon that can run this, it's the pairing of the accelerator and gigs of RAM that is the key.

Re: Transformer architecture optimized for Apple Silicon

#59

Post by windowsrookie »

btown wrote: post #21

I find it fascinating that this was released after the June 2022 launch of the M2 chipset and line of products, and yet Apple had no desire to show relative performance of M2 vs. M1 here - even in the simultaneous announcement here: https://machinelearning.apple.com/research/neural-engine-tra... It's fascinating to me that at least one of two things is true: either (a) Apple has lost its ability to coordinate "hype"…

It's my understanding the the performance difference is very small between the M1 and M2. So small that Apple is still selling new M1 Macs today.

The M2 added more GPU cores and an advanced media decoder.

Re: Transformer architecture optimized for Apple Silicon

#60

Post by jorvi »

davnicwil wrote: post #19
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

Siri was supposed to become locally processed but sometimes can’t even set a timer because of a connection time-out. Or she’ll use her fancy ML speech recognition model to turn “set a timer for 3 minutes 10 seconds” into “search for trinity tensor”.

So much for an “unfathomable moat”.

Post reply on HN