Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

111–120 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#111

Post by qumpis »

garblegarble wrote: post #90
s1k3s wrote: post #39

Earlier quoted context omitted.

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as the…

Offtopic, but for what purpose are you running llms locally (especially everyday)? My understanding was that the prompting requires to make them work at all was too great.

Re: Transformer architecture optimized for Apple Silicon

#112

Post by qumpis »

monocasa wrote: post #66
s1k3s wrote: post #47

Earlier quoted context omitted.

Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?

The rumor I've heard is that GPT4 didn't meaningfully increase the parameter count versus GPT3.5, but instead focused on training and structural improvements.

Well the inference time of gpt4 seems to be far greater than gpt3, so it could hint a difference in parameters count.

Re: Transformer architecture optimized for Apple Silicon

#113

Post by bitL »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

That's highly unlikely. I think most people can't fathom how huge is the scale of GPT-4 and what computing power it requires. Even if 10x optimizations, 10x performance and 10x memory (highly unlikely in 5 years) that's not going to be sufficient to run it locally.

Re: Transformer architecture optimized for Apple Silicon

#114

Post by gumby »

arthurcolle wrote: post #84
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]...

> Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today.

Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn't ask me :-).

John Giannandrea has run Apple's AI effort for the past five or six years. He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple. For all I know the Siri team isn't even part of his remit.

You can never look into Apple (even if you work there) so one can only speculate based on what visible signs appear. But Siri isn't one of them.

Re: Transformer architecture optimized for Apple Silicon

#115

Post by jazzyjackson »

itg wrote: post #82
api wrote: post #35

Earlier quoted context omitted.

Apple M series chips are already some of the best chips for running your own models locally, especially if you want a laptop or small form factor machine instead of a big gaming desktop with a big GPU. Apple is really getting a niche here for machines to run models locally. That’s pretty powerful.

I would not say they are the best chips if you want to run locally. They are ok although a bit slow compared to desktop gpus for inference.

What desktop GPUs would you recommend for the home user with access to 20+ GB of VRAM? I can get a Mac Studio with 128GB of shared RAM for about 5k.

Re: Transformer architecture optimized for Apple Silicon

#116

Post by jazzyjackson »

norgie wrote: post #98
guestbest wrote: post #43

Earlier quoted context omitted.

Apple will never have their tech run locally for AI anymore than they allow Siri to search the internet offline or while driving

How would anyone search the internet offline?

Simple, you make a local copy of the internet.

Re: Transformer architecture optimized for Apple Silicon

#117

Post by richardw »

arthurcolle wrote: post #84
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

If you’re the exec in charge of this at Apple, what are you doing right now. It’s not watching OpenAI become what Siri could be unless you have something to do about it. Your motivation levels are extremely high.

Re: Transformer architecture optimized for Apple Silicon

#119

Post by guestbest »

norgie wrote: post #98
guestbest wrote: post #43

Earlier quoted context omitted.

Apple will never have their tech run locally for AI anymore than they allow Siri to search the internet offline or while driving

How would anyone search the internet offline?

You can’t . That’s the joke. If you use car play and ask Siri to find a local destination or a celebrity birthday, the functionality will be refused since the driver is assumed to be operating the car and thus blocks internet search, even though it will search in Apple Maps which obviously isn’t cached on the device

Re: Transformer architecture optimized for Apple Silicon

#120

Post by pornel »

s1k3s wrote: post #39
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

Hundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds. It's also likely that memory requirements will be shrunk through software and ML improvements.
Post reply on HN