Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

41–50 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#41

Post by coldtea »

btown wrote: post #21

I find it fascinating that this was released after the June 2022 launch of the M2 chipset and line of products, and yet Apple had no desire to show relative performance of M2 vs. M1 here - even in the simultaneous announcement here: https://machinelearning.apple.com/research/neural-engine-tra... It's fascinating to me that at least one of two things is true: either (a) Apple has lost its ability to coordinate "hype"…

>I find it fascinating that this was released after the June 2022 launch of the M2 chipset and line of products

And this was put out in Aug 2022. It's very propable that the team worked on it and tested it just on M1, and M2 was kept under wraps in different teams working on it until the announcement. So they just wrote the annoucement on the CPUs they worked on - and since it's not for a commercial product, Apple didn't care to optimize marketing anyway.

Re: Transformer architecture optimized for Apple Silicon

#42

Post by smoldesu »

endisneigh wrote: post #28
smoldesu wrote: post #22

Earlier quoted context omitted.

I mean, it's possible to load LLMs onto a smartphone today. The utility is limited though, and if iPhone 19 still has arbitrary application memory requirements then I think it's safe to say they won't be using the most complete models. At the end of the day, OpenAI's "huge-ass LLM as a service" will probably be relevant longer than you think it will be. Local inferencing might be able to do simpler stuff (eg. compose…

I never said anything about openai not being relevant. the point is scale and good enough. gpt-4 is already good enough for most reasonable use cases, notably siri optimization. 5 years is a long time. 5 years ago llms didn't even exist.

> 5 years ago llms didn't even exist

Almost 5 years ago TalkToTransformer did 80% of ChatGPT's job with 0% of the hype. Once people realize just how glacial all this stuff moves, I think the honeymoon phase will be over.

Re: Transformer architecture optimized for Apple Silicon

#43

Post by guestbest »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Apple will never have their tech run locally for AI anymore than they allow Siri to search the internet offline or while driving

Re: Transformer architecture optimized for Apple Silicon

#44

Post by endisneigh »

s1k3s wrote: post #39
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude.

at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.

Re: Transformer architecture optimized for Apple Silicon

#45

Post by ibejoeb »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

OpenAI's relationship with Microsoft pretty much makes it big tech. I think you have a point as long Apple keeps innovating its hardware, but it's not really a David vs Goliath situation.

Re: Transformer architecture optimized for Apple Silicon

#46

Post by eternalban »

leetrout wrote: post #17
tinyhouse wrote: post #12

This is great. I cannot wait to try it on my laptop as I like to do dev locally. But I don't understand the development part - besides on device, how would you deploy this on a server let's say given Apple serves is not something cloud providers provide.

Phones not servers.

Edge not center.

Re: Transformer architecture optimized for Apple Silicon

#47

Post by s1k3s »

endisneigh wrote: post #44
s1k3s wrote: post #39

Earlier quoted context omitted.

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.

Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?

Re: Transformer architecture optimized for Apple Silicon

#48

Post by endisneigh »

smoldesu wrote: post #42
endisneigh wrote: post #28

Earlier quoted context omitted.

I never said anything about openai not being relevant. the point is scale and good enough. gpt-4 is already good enough for most reasonable use cases, notably siri optimization. 5 years is a long time. 5 years ago llms didn't even exist.

> 5 years ago llms didn't even exist Almost 5 years ago TalkToTransformer did 80% of ChatGPT's job with 0% of the hype. Once people realize just how glacial all this stuff moves, I think the honeymoon phase will be over.

right, that's why I mentioned performance at the current level at far less compute, and not necessarily better performance.

Re: Transformer architecture optimized for Apple Silicon

#49

Post by brundolf »

endisneigh wrote: post #36
brookst wrote: post #33

Earlier quoted context omitted.

What other services does Apple charge a subscription fee for despite having no opex?

apple fitness I imagine has a pretty small opex (sure they pay trainers, but that doesn't scale linearly with subscriptions. the videos themselves are limited in scope and could be cached pretty trivially. the actual fitness tracking happens on device), same with apple arcade which I highly doubt has any additional expensive over the app store in general. I can see it - "Siri+", pay $5 a month for fine tuned model up…

You don't need a subscription to track and view your data (I've got an apple watch and no subscription). I think all the subscription gets you are the training courses (videos, but also live events and stuff)

Re: Transformer architecture optimized for Apple Silicon

#50

Post by endisneigh »

s1k3s wrote: post #47
endisneigh wrote: post #44

Earlier quoted context omitted.

yes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.

Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?

that's a complicated question to answer. what I'd say is that more parameters makes the model more robust, but there are diminishing returns. optimizations are under way
Post reply on HN