Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

321–330 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#321

Post by sroussey »

LoganDark wrote: post #142
afavour wrote: post #99

Earlier quoted context omitted.

“Yes Siri is garbage but it doesn’t matter because Apple are probably working on a top secret new thing that will blow everyone else out of the water” You might be right but there’s no real evidence for it yet.

Hasn't Siri remained relatively unchanged for over a decade? I think the most recent innovation was just bringing it to Intel Macs.

Most people didn’t notice, but dictation now runs in a hybrid mode — both on device and via server. So you get some (maybe poor) results at first (only results if offline but better than nothing) that get rewritten after a couple seconds from server.

For Siri itself, I think it runs locally now.

Re: Transformer architecture optimized for Apple Silicon

#322

Post by mark212 »

agentcoops wrote: post #307
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

Honestly, it speaks mountains to Apple as an organisation that they did not invest all of their resources into Siri, especially when you consider how Amazon invested in Alexa etc. Everyone has long understood that the "personal assistant" represents really the pinnacle of not just what AI but the personal computer should be able to offer -- the mistake this past decade was thinking that had to do with then current ML…

Amazon's investment in Alexa is a perfect example of why Apple didn't (and shouldn't have) invested all of their resources into Siri. The Alexa team is getting seriously gutted in these rounds of downsizing [1], or as CNBC puts it "the team behind the technology was a prime target of the largest layoffs in the company’s history."

What they have been invested heavily in is the Apple Neural Engine ("ANE"), special silicon right on the SoC to handle ML / AI code. Optimize on a server, then run the model on your iPhone or probably soon, your Apple Watch.

WWDC this year is going to be very, very important.

[1] https://www.cnbc.com/2023/01/06/amazon-fully-committed-to-al...

Re: Transformer architecture optimized for Apple Silicon

#323

Post by waffletower »

Not sure why Apple is throwing this code over the fence, (or catapulting it over the balustrade -- to continue the castle metaphor used here) and not also selling server form factor Apple silicon based devices for data centers. They left out an "issues" pane on the github repo, so it is intended to be quite the unidirectional act. I am not sure that Apple will remain proportionally huge with respect to other big tech while they squander their development on boutique mass market products, while ignoring the vast growth they could achieve if they expanded into first class cloud computing markets.

Re: Transformer architecture optimized for Apple Silicon

#324

Post by Tagbert »

gcanyon wrote: post #178
epistasis wrote: post #130

Earlier quoted context omitted.

Me: Hey Siri, call Toyota. Siri: I'm sorry, that contact is not in your list Me: Siri, what is the number for Toyota. Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go] Me: Siri, call Toyota Siri: I'm sorry, you do not have that contact number. Me: &$@@&/&&/&!!!

The perfectly obvious use cases Siri overlooks are maddeningly numerous. My go to example was "text this photo to " For years I checked each new iOS release to see if that was enabled, and for close to ten years, no. I'm not sure whether it was the most recent, or the one before (I gave up somewhere along the way) but now you can (finally) do it.

"HeySiri, send this photo to name in messages." "Ready to send" "Send it" "It's sent"

done

Perhaps the verb "text" is unclear to Siri?

Re: Transformer architecture optimized for Apple Silicon

#325

Post by anukin »

aunty_helen wrote: post #294
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Honestly, I've looked at shorting MS due to openai. Their technology (LLMs), or the secret sauce, can easily be stolen just by the process of putting that tech out there. Have a look at Alpaca, FB made it, someone leaked the weights and now there's a dataset of training it for only a few hundred dollars that can beat openai at its best. Not everyone needs to employ a PhD for doing customer service, in the same way no…

TBH alpaca is nowhere near to chatgpt or even gpt models in its capability. I have been playing around with lama for a while now and even chain of prompts, the quality of answers are really bad. I had thought that Facebook would be having the ability and competence to produce a LLM which can outclass open ai

Re: Transformer architecture optimized for Apple Silicon

#326

Post by pb7 »

sparrc wrote: post #211
inferyes wrote: post #208

Earlier quoted context omitted.

> I'm not so sure. When is the last time apple came out something groundbreaking? M1 chips?

An ARM cpu is groundbreaking? I don't think anyone outside the tech world has even noticed.

Name something that you find groundbreaking and I too can reduce it down to something mundane.

Leaps in performance across all metrics in an existing thing is groundbreaking, especially when it’s just the beginning.

On a practical level, it feels pretty groundbreaking to me when I go back to use my previously top of the line 16” MBP from a year before. I suspect you haven’t had the pleasure of using an M-series computer.

Re: Transformer architecture optimized for Apple Silicon

#327

Post by anentropic »

lr1970 wrote: post #154

The github repo has only 5 commits in it and the last one is August 9, 2022. It looks abandoned.

It's a reference implementation, other people are supposed to pick it up and apply the techniques to their own work

I would love to see some links to any where it has been used?

So far the most promising codebases for running LLMs on Mac have been the 'cpp' reimplementations, which ditch Pytorch and run on the CPU, using other tricks to fit the model into available RAM.

Re: Transformer architecture optimized for Apple Silicon

#328

Post by josephjrobison »

davnicwil wrote: post #19
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

In addition, Apple is notoriously slower at adopting the most cutting edge technologies but historically they tend to consider things very thoughtfully and rolls them out in beautiful launches (not always).

Examples would be the fact that Samsung almost always beats them to the punch on camera technology and other whiz-bang features, but Apple eventually adopts to much mass market consumer acclaim and groans from Android techies. Another example is they’re just now considering touch screen on laptops.

A counter example is the TouchBar - which was “innovative” but many didn’t like.

Re: Transformer architecture optimized for Apple Silicon

#329

Post by brookst »

smoldesu wrote: post #319
brookst wrote: post #311

Earlier quoted context omitted.

ML inference is not generally FP32 anymore. I was going off of the TOPs numbers for ML from a few sources, which generally agree M2 is about 22 TOPS and 4080 (desktop) is about 50. But in any event, yes, that was my point. UMA is a huge advantage, the GPU itself is too weak to be serious. But it’s a lot easier to drop a dramatically beefier GPU into a new design than it is to update the entire platform for UMA. Apple…

> whether tbey pursue it or not remains to be seen. Pursue what though? UMA is cool, but kinda meaningless if the majority of Macbooks are min-spec. That leaves you with 4-5gb of VRAM, assuming you've left nothing open. What is Apple going to do with that UMA that other manufacturers cannot? It's certainly nice that 128gb Macs exist for models that might be too big to otherwise load into memory. It's useless for prod…

> Pursue what though?

A Mac variant that trades CPU cores for GPU/ML cores while having 192GB+ of UMA memory.

> I struggle to imagine the “opportunities”

Two of them: 1) academic / R&D compute, where people could have at least A6000 class GPU on the desktop, and 2) cloud inference servers, probably for Apple’s own services.

I’m not saying they will or should do those things, just that the apple silicon arch is well positioned if they choose to. Bolting on exponentially better GPU is not especially difficult, and they’ve got an OS that would bring existing apps and libraries right over.

Look at it this way: is there a path to UMA on Windows / Linux? If not, those systems will always duplicate RAM and require users to decide in advance whether to allocate RAM budget to OS or ML.

Re: Transformer architecture optimized for Apple Silicon

#330

Post by parker_mountain »

ohgodplsno wrote: post #245
parker_mountain wrote: post #156

Earlier quoted context omitted.

Homepods have U1 chips. They can position airtags within your house. You can put an airtag under, or adjacent to, a lamp and link it to a smart switch. Your apple watch has gesture detection (still in beta, tbf), as well as U1 and real time positioning capabilities. Everything required for apple to know not only where you are, but which way your hand is pointing, as well as where "smart devices" are in your house is…

$25 for an Airtag, plus say $20 for a smart switch. Plus $200 for a HomePod. To turn a single lamp on/off by pointing at it, something that nobody ever wants to do (alright, once for the cool factor). If someone wanted to overpay for useless features, they can already go for a Philips Hue. I can turn off my lamps from anywhere in the world using a $10 Tuta ZigBee bridge and a $8 LIDL light.

I think that this is one of those things that sounds like you'd never do it, but...

I've put zigbee switches into every wall switch in my house and now when I stay in hotels, I forget to turn off the lights before getting into bed.

>something that nobody ever wants to do

I think that this gesture, if it works well, is something that will be a killer app for smarthomes. The other point of friction, which is interop, has been more or less solved by thread/matter. Homepods are $100, btw. Not to mention what happens if apple integrates u1 into a smart switch/bulb, which is something that the thread protocol allows for.

Post reply on HN