Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

221–230 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#221

Post by post-it »

parker_mountain wrote: post #88
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

It's probably not glasses. Nobody likes wearing glasses. Every attempt at making tech glasses has failed. It'll probably be the Apple Watch.

Re: Transformer architecture optimized for Apple Silicon

#222

Post by illegalmemory »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I also believe this is going to the case. The past decade apple has positioned itself as a privacy focused + good hardware company. The next is going to be offline AI + privacy + good hardware & I am already sold to the idea.

Re: Transformer architecture optimized for Apple Silicon

#223

Post by PopePompus »

LoganDark wrote: post #142
afavour wrote: post #99

Earlier quoted context omitted.

“Yes Siri is garbage but it doesn’t matter because Apple are probably working on a top secret new thing that will blow everyone else out of the water” You might be right but there’s no real evidence for it yet.

Hasn't Siri remained relatively unchanged for over a decade? I think the most recent innovation was just bringing it to Intel Macs.

Yes, it's been stagnant for so very long that I, and I suspect most users, have learned the very few things that it can do reliably, and no longer even try to do something more sophisticated. If it became more capable, it might take a long while for most users to even notice, because we have stopped hoping it will pleasantly surprise us.

Re: Transformer architecture optimized for Apple Silicon

#224

Post by fullsend »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Definitely possible. A cool idea. But that’s also a lot of credit for “big tech”. Meta, Google, Netflix, and others have produced almost nothing cool for a decade. Apple is one of the few behemoths seemingly capable of actually getting any real work done, so you’re right they’ll be a big contender. Smaller players have less capital, but a lot less blockers too. They can take real risks, move faster, and don’t have all the bottlenecks that come with having 50,000 employees.

Re: Transformer architecture optimized for Apple Silicon

#225

Post by mrtksn »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

Hyper casual games are around 300Mb these days, proper AAA games are multiple GB. People still download those, as you can tell by the billions of dollars they make.

The problem with OpenAI's business model is that it's actually quite expensive for them to maintain centralised processing. With Apple, there are billions of very powerful computers deployed to users and these computers mostly stay idle apart from occasionally running some bloated JS to show a button ar something. If Apple manages to run a good enough model on device with acceptable performance and energy impact, then suddenly OpenAI and Microsoft will be just burning away money with no expectation of recouping if they provide the service for free, if they make it paid they will be making money in a niche.

Re: Transformer architecture optimized for Apple Silicon

#227

Post by j45 »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

This sounds great, but why can’t Siri still not understand me.

Maybe Apple has the ultimate

Re: Transformer architecture optimized for Apple Silicon

#228

Post by nicky0 »

parker_mountain wrote: post #88
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

As an mere user this is what I had assumed. Siri has been stagnant and buggy for years; it makes sense they are working on a big replacement instead of putting resource into incremntal improvements on what seems to be a weak foundation.

Re: Transformer architecture optimized for Apple Silicon

#229

Post by singularity2001 »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

Yes, by the time iphones can smoothly run llama 30G, the state of the art gpt-x will probably be a terabyte. Skynet will forever live in the cloud with just assistant agents living on the devices.
Post reply on HN