Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

71–80 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#71

Post by 0xDEF »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

>openai is doing great work and is serious competition, but I think many underestimate big tech

OpenAI is 49% owned by Microsoft. OpenAI is big tech.

Re: Transformer architecture optimized for Apple Silicon

#72

Post by highwaylights »

brokencode wrote: post #68
ibejoeb wrote: post #45

Earlier quoted context omitted.

OpenAI's relationship with Microsoft pretty much makes it big tech. I think you have a point as long Apple keeps innovating its hardware, but it's not really a David vs Goliath situation.

Nvidia is also no David to Apple’s Goliath. They are both Goliaths who will most likely be battling to build the best hardware for LLMs in the near future.

It’s an exciting time.

I think people are very prematurely counting AMD and even Intel out of this race considering the pace of change.

Re: Transformer architecture optimized for Apple Silicon

#73

Post by dev_tty01 »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I'm expecting the Mac Pro to have AI/ML massive GPU modules and/or specialized ML dedicated compute available as add-ons for in-house training. For a not-so-small fee of course. They have all the pieces, it just needs to be brought to bear on the ML training market.

Re: Transformer architecture optimized for Apple Silicon

#74

Post by smoldesu »

api wrote: post #35
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Apple M series chips are already some of the best chips for running your own models locally, especially if you want a laptop or small form factor machine instead of a big gaming desktop with a big GPU. Apple is really getting a niche here for machines to run models locally. That’s pretty powerful.

> Apple is really getting a niche here for machines to run models locally.

I... don't agree. They have an acceleration API and a large install-base, but all of these models have run just fine on traditional hardware. GPT-Neo, Stable Diffusion and LLaMa can all be used and accelerated without Apple Silicon.

Powerful, maybe. But not really unique, just putting up table stakes.

Re: Transformer architecture optimized for Apple Silicon

#75

Post by smoldesu »

tinyhouse wrote: post #12

This is great. I cannot wait to try it on my laptop as I like to do dev locally. But I don't understand the development part - besides on device, how would you deploy this on a server let's say given Apple serves is not something cloud providers provide.

There are acceleration options for ARM, either Microsoft's ONNX or proprietary "Ampere-optimized" libtorch are both good options in my experience.

Re: Transformer architecture optimized for Apple Silicon

#76

Post by acdha »

As a newbie to this space, I see this mentioning PyTorch. I was looking at Whisper earlier today and, somewhat impressively, was reminded that my M1 Pro has a CPU fan. Is it realistic to think it would be a modest amount of work to install this in my local venv and use the NPU instead?

Re: Transformer architecture optimized for Apple Silicon

#77

Post by fnordpiglet »

highwaylights wrote: post #70
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

Maybe, but Apple doesn’t have a search to rival Google (or even an assistant, given the state of Siri). Focusing on privacy and on-device learning is great, but when the strength of these models is in consuming all the data they can hoover up your motive is at odds with your philosophy.

I think this is wrong because you can always build a search engine - it’s not hard - and you can supplement the learned models with cryptographically secured private data to build a private personalized model. In fact this seems in all ways superior to any alternative, and strikes at the heart of the Google spy ware ad spam business model.

And seriously, did you really try to assert the Google whatever it is has outperformed Siri? I’d love to meet someone that uses it so I can verify this statement. But with the advent of Chatgpt Alexa, Siri, and Google whatever all look like freshman projects. The next generation of assistants have just begun, and I assure you, the state of the past has nothing to do with what comes next. OpenAI hit a giant reset button across an awful lot of stuff.

Re: Transformer architecture optimized for Apple Silicon

#78

Post by threeseed »

highwaylights wrote: post #70
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

Maybe, but Apple doesn’t have a search to rival Google (or even an assistant, given the state of Siri). Focusing on privacy and on-device learning is great, but when the strength of these models is in consuming all the data they can hoover up your motive is at odds with your philosophy.

They do have a search engine though with Siri Suggestions. Which has a user base of ~1.5b+ people.

We only see the tip of the iceberg so who knows what else it is capable of.

Re: Transformer architecture optimized for Apple Silicon

#79

Post by FloatArtifact »

sedatk wrote: post #64
brookst wrote: post #37

Earlier quoted context omitted.

> it will be convenience rather than privacy which is the deciding factor This. Nobody really cares about local processing for privacy. Even those that claim to often don't mean it. Remember the total freakout over Apple's proposed local, privacy-preserving processing to detect CSAM before uploading it to iCloud? The consensus seemed to be that secret, opaque, and un-auditable cloud-based scanning was much preferable…

Local CSAM scanning wasn't an open book either. Also, it wasn't mutually exclusive with cloud scanning.

However, there was enough of an outcrying by the community and my experts that it was pulled. Therefore end users do care about privacy when they understand the implications.

After all, it's hard to care when you don't have power as an end user too act on those feelings.

Re: Transformer architecture optimized for Apple Silicon

#80

Post by coolspot »

fnordpiglet wrote: post #77
highwaylights wrote: post #70

Earlier quoted context omitted.

Maybe, but Apple doesn’t have a search to rival Google (or even an assistant, given the state of Siri). Focusing on privacy and on-device learning is great, but when the strength of these models is in consuming all the data they can hoover up your motive is at odds with your philosophy.

I think this is wrong because you can always build a search engine - it’s not hard - and you can supplement the learned models with cryptographically secured private data to build a private personalized model. In fact this seems in all ways superior to any alternative, and strikes at the heart of the Google spy ware ad spam business model. And seriously, did you really try to assert the Google whatever it is has outp…

> And seriously, did you really try to assert the Google whatever it is has outperformed Siri?

Anything outperforms Siri. Google Assistant is head and shoulders above Alexa and Siri.

Post reply on HN