Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

291–300 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#291

Post by mgdante »

tlrobinson wrote: post #272
ohgodplsno wrote: post #245

Earlier quoted context omitted.

$25 for an Airtag, plus say $20 for a smart switch. Plus $200 for a HomePod. To turn a single lamp on/off by pointing at it, something that nobody ever wants to do (alright, once for the cool factor). If someone wanted to overpay for useless features, they can already go for a Philips Hue. I can turn off my lamps from anywhere in the world using a $10 Tuta ZigBee bridge and a $8 LIDL light.

You sir, are not Apple’s target market.

Seems like Apple's target market is very narrow given homepod's weak sales

Re: Transformer architecture optimized for Apple Silicon

#292

Post by aunty_helen »

hardware2win wrote: post #288
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

OpenAI and MSFT couldnt figure it out and are burning milions of USD on compute, but Apple will make it run on phone in 5 years? Crazy bet, what makes you think that? You cannot optimize infinitely. Raytracing probably had decades of ppl trying to make it run fast and yet even today you need strong hardware

Have a look at what the opensource community is up to. They're reducing models down to the meer billions of nodes and then using chatgpt to train their model accuracy. The results are suprising.

The reason openai have had to rush out plugins is due to software like langchain coming in at meteoric speed.

Things are moving on a day to day basis in the ai sphere at the moment.

Re: Transformer architecture optimized for Apple Silicon

#293

Post by pantulis »

parker_mountain wrote: post #88
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

> apple doesn't get the full advantage of tons of user data from the wild.

Do you think OpenAI did? Perhaps Apple doesn't have the vast amount of data that Google has, but if OpenAI managed to have like 200 wikipedia-sized corpuses of different textual data for their English GPT models, that's certainly not out of reach of Apple.

Re: Transformer architecture optimized for Apple Silicon

#294

Post by aunty_helen »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Honestly, I've looked at shorting MS due to openai.

Their technology (LLMs), or the secret sauce, can easily be stolen just by the process of putting that tech out there.

Have a look at Alpaca, FB made it, someone leaked the weights and now there's a dataset of training it for only a few hundred dollars that can beat openai at its best.

Not everyone needs to employ a PhD for doing customer service, in the same way not everyone needs GPT5 for answering support queries.

Their business model is leaking away from them.

Re: Transformer architecture optimized for Apple Silicon

#295

Post by mirekrusin »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

When you'll be able to run gpt4 on iphone 19, there will be gpt27 that doesn't run on it.

Re: Transformer architecture optimized for Apple Silicon

#296

Post by crooked-v »

mlboss wrote: post #52
Gigachad wrote: post #27

Earlier quoted context omitted.

GPT-3 was said to require something like 150gb of VRAM. I don't see that gap being bridged in phones within 2 years.

LLaMA already runs on M2 and is comparable to gpt 3

I gave a try at Alpaca-LoRA with tloen's tuning and it definitely feels in reach of GPT 3. Not quite as good, but some of that may just be in whatever's going on with OpenAI's hidden prompts encouraging lots of text out of the model.

Re: Transformer architecture optimized for Apple Silicon

#298

Post by michelb »

qgin wrote: post #104

In regards to LLMs there is a collision between Apple's extremely good chip design capabilities and Apple's insistence that Siri never says anything that isn't 100% scripted and 100% certain to not bad. Up until now, they've chosen to limit Siri functionality rather than leave anything to chance. LLMs will absolutely be able to run locally, but whether Apple will be able to stop worrying and love the model remains to…

Pretty sure Siri is getting a complete overhaul since it's a pita to add anything to it right now.

Re: Transformer architecture optimized for Apple Silicon

#299

Post by zitterbewegung »

davnicwil wrote: post #19
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

They might not even require the need to train. There are many open implementations of models that don’t have an academic requirement. Right now you can use tensorflow.js which runs on the gpu using webgl of a phone. If you build the API out developers will use it.

Re: Transformer architecture optimized for Apple Silicon

#300

Post by KuzMenachem »

ronyfadel wrote: post #197
ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

That is not even taking into consideration that shipping your weights to the client is akin to giving your product away.

You can put some obstacles (https://developer.apple.com/documentation/coreml/generating_...) in the way though.
Post reply on HN