Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

201–210 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#201

Post by CGamesPlay »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

Research developments are already showing that our models are woefully inefficient in their current state (compare the performance of GPT-3 140B against Alpaca 30B). Not only will hardware get better, the minimum model sizes for good inference will become smaller in the future.

Re: Transformer architecture optimized for Apple Silicon

#202

Post by ThorsBane »

ThorsBane wrote: post #168
calvinmorrison wrote: post #166

Earlier quoted context omitted.

why does this matter when we have cloud computing and high speed mobile internet? why does it need to be local

Because it's more seamless and then integrates into the Apple hardware ecosystem due to AirDrop and other integrations.

And also it represents cool and significant hardware advances. That’s always sexy, and on-device AI will expand device capabilities.

Imagine Google Sonar on Pixel phone with Pixel ear buds and more sensors and ChatGPT onboard.

Re: Transformer architecture optimized for Apple Silicon

#203

Post by [deleted] »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

[deleted]

Re: Transformer architecture optimized for Apple Silicon

#204

Post by aix1 »

ronyfadel wrote: post #197
ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

That is not even taking into consideration that shipping your weights to the client is akin to giving your product away.

Wait, how is that fundamentally different from shipping binary code? Isn't that also akin to "giving your product away"?

Re: Transformer architecture optimized for Apple Silicon

#205

Post by sparrc »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I'm not so sure. When is the last time apple came out something groundbreaking? Has there been anything since Tim Cook took over?

Siri is still extremely basic and barely usable for anything besides starting a kitchen timer.

Re: Transformer architecture optimized for Apple Silicon

#206

Post by ronyfadel »

aix1 wrote: post #204
ronyfadel wrote: post #197

Earlier quoted context omitted.

That is not even taking into consideration that shipping your weights to the client is akin to giving your product away.

Wait, how is that fundamentally different from shipping binary code? Isn't that also akin to "giving your product away"?

You can easily pirate an ML model, host it and provide a backend service that uses it, and no one would suspect anything.

I don't think you can pirate e.g. the Facebook app binary and make it your own.

Re: Transformer architecture optimized for Apple Silicon

#207

Post by Dalewyn »

sparrc wrote: post #205
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I'm not so sure. When is the last time apple came out something groundbreaking? Has there been anything since Tim Cook took over? Siri is still extremely basic and barely usable for anything besides starting a kitchen timer.

Apple Silicon.

Re: Transformer architecture optimized for Apple Silicon

#208

Post by inferyes »

sparrc wrote: post #205
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I'm not so sure. When is the last time apple came out something groundbreaking? Has there been anything since Tim Cook took over? Siri is still extremely basic and barely usable for anything besides starting a kitchen timer.

> I'm not so sure. When is the last time apple came out something groundbreaking?

M1 chips?

Re: Transformer architecture optimized for Apple Silicon

#209

Post by abudabi123 »

duped wrote: post #128
gumby wrote: post #114

Earlier quoted context omitted.

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn…

Jobs has been dead for over ten years

The cinema industry know how to squeeze a billion dollar a year out of Apple. Do the Apple or Sony ”social engineers“ have total view? When Amazon was sky rocketing and Blue Origin had not reached orbit a niche producer from the cinema industry had Jeff B paying a lot although he might have backed out of that deal like giving up on the Android phone with a lot of cameras.

Re: Transformer architecture optimized for Apple Silicon

#210

Post by atleastoptimal »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Local inference is a pipe dream. There's no way any of these companies will allow consumers to interact with their flagship models on anything but a black box api.
Post reply on HN