Transformer architecture optimized for Apple Silicon

Post by shmoogy » Thu, Mar 23, 2023, 11:16 PM UTC

endisneigh wrote: post #28

smoldesu wrote: post #22

Earlier quoted context omitted.

I mean, it's possible to load LLMs onto a smartphone today. The utility is limited though, and if iPhone 19 still has arbitrary application memory requirements then I think it's safe to say they won't be using the most complete models. At the end of the day, OpenAI's "huge-ass LLM as a service" will probably be relevant longer than you think it will be. Local inferencing might be able to do simpler stuff (eg. compose…

I never said anything about openai not being relevant. the point is scale and good enough. gpt-4 is already good enough for most reasonable use cases, notably siri optimization. 5 years is a long time. 5 years ago llms didn't even exist.

Post by brookst » Thu, Mar 23, 2023, 11:17 PM UTC

endisneigh wrote: post #30

armchairhacker wrote: post #26

Earlier quoted context omitted.

This would be awesome. I don't like having to rely on closed-source cloud services for LLMs. But what incentive does Big Tech have? Especially considering that presumably they could monetize the cloud services easier. It's not like most consumers care either. I expect smaller companies like Stable Diffusion and grant/govt-funded research to bring cheap, local inference. There's definitely a lot of demand. But is ther…

i'd say apple is one of the few companies that could charge people for an "AI embedded" 128GB module, allow people to pay for a subscription in order to "access", yet have the inference happen locally. one hypothetical is that you need an icloud subscription, a token is retrieved from apple, the token "unlocks" your AI module, on the phone and allows you to do the inference. in this way apple could charge monthly for…

Post by api » Thu, Mar 23, 2023, 11:18 PM UTC

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Post by brookst » Thu, Mar 23, 2023, 11:19 PM UTC

rcarr wrote: post #29

great_psy wrote: post #5

Maybe apple will have a bigger effect on ai adoption than any other company. Local inference is huge for anything that requires even a little bit of privacy.

Convenience will be the biggest factor. Whoever makes it easier for the end consumer to get what they want wins. It's why ChatGPT made such a big splash in comparison to all the other AI models which were also impressive. Local inference may play a part in that if it's quicker but if the last twenty years are anything to go by it will be convenience rather than privacy which is the deciding factor. And Apple do tend…

Post by s1k3s » Thu, Mar 23, 2023, 11:21 PM UTC

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…