Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

11–20 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#11

Post by bottlepalm »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Maybe we should launch 100 of them out into space in different directions. Very low mass means we should be able to push it to a pretty high velocity.

Re: Transformer architecture optimized for Apple Silicon

#13

Post by au8er »

While the github contains the code, the article describing the optimisations are here: https://machinelearning.apple.com/research/neural-engine-tra....

TL;DR: execution of pytorch models on apple's neural engine and standard data-oriented optimisations (changing matrix layout, chunking to optimise temporal cache locality, and minimising redundant memory copies)

Re: Transformer architecture optimized for Apple Silicon

#15

Post by simonh »

hannofcart wrote: post #3

As someone entirely at sea with the rapid pace of development in this sphere: 1. Is this a new LLM from Apple? 2. Is this a way to optimize running LLMs like Llama locally on M1 macs? 3. Something else altogether?

It this provides optimised hardware acceleration on Apple Silicon for low level machine learning libraries.

PyTorch is supported for example, it’s a machine learning library with GPU acceleration that’s been around for 6 years now. It’s used in a few commercial projects, including Tesla Autopilot. It can be used for natural language processing, image manipulation, and possibly to build an LLM I suppose, but as a low level library it just gives the base tech to build such systems from.

Re: Transformer architecture optimized for Apple Silicon

#17

Post by leetrout »

tinyhouse wrote: post #12

This is great. I cannot wait to try it on my laptop as I like to do dev locally. But I don't understand the development part - besides on device, how would you deploy this on a server let's say given Apple serves is not something cloud providers provide.

Phones not servers.

Re: Transformer architecture optimized for Apple Silicon

#18

Post by ll_mama »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

5 years? Bit of a long stretch with how much focus is going into AI. I’d be surprised if it’s not possible within two iterations of the new iPhone.

Re: Transformer architecture optimized for Apple Silicon

#19

Post by davnicwil »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices.

We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waiting in the wings.

Not only that, but this model of 'wait in the wings and pounce when the tech is really there and absolutely nail it' is Apple's forté, aligned completely with the cultural and strategic DNA of the company.

Re: Transformer architecture optimized for Apple Silicon

#20

Post by hokkos »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

But users will need to store multiples TB of data to run it ?
Post reply on HN