Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

121–130 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#121
post #60

Earlier quoted context omitted.

Siri was supposed to become locally processed but sometimes can’t even set a timer because of a connection time-out. Or she’ll use her fancy ML speech recognition model to turn “set a timer for 3 minutes 10 seconds” into “search for trinity tensor”. So much for an “unfathomable moat”.

Siri is so bad it needs to be scrapped and rethought from the ground up (it can’t even give me the time when internet is down at my rural house). No one at Apple who wants a career there would dare propose that. The switch to a LLM architecture could be the perfect transition point for this.

If Siri can't give you the time without the internet, I think you need to update iOS.

There are definitely two tiers of Siri queries. There are queries like "set the brightness to 10%" or "set a timer for 5 minutes" which absolutely and consistently work without internet, and have for several years, and if you're legitimately having a different experience then its possible a cosmic ray hit your iPhone (or, realistically, you're running into a strange and rare bug which is not indicative of the general experience and will be fixed in three to five years or maybe never). There are also queries like "create a new note" (from the Notes app) which should be able to work offline, but don't. And, naturally, there are queries like "when did Resident Evil 4 come out" which wouldn't reasonably work without internet (but, if you're curious, she does get it right).

In other words, Siri clearly does some local guessing as to whether she can answer a query without the internet, and some queries appear to be miscategorized into the second bucket. My leading theory on why this happens, which may be incorrect, but it seems like: if an app has any Siri functionality which requires the internet to answer, all of the queries which are responded to by that app have to require the internet. It doesn't matter where the processing ends up actually happening to respond to the query, it just shuts the query down. Its weird, but its consistent with the behavior I've seen.

The more important point: Siri's real and weird limitations don't seem to have much to do with limitations in local processing. They've said that the speech interpretation all happens on-device. They encrypt practically all of your data that does get shipped to their servers, so a query like "Open the note titled 'Hello World'" is probably also being processed on-device. But: Siri still requires the internet for that query. That doesn't seem like a significant limitation with their ML algorithms or silicon or anything meaningful; it seems like just a case of dumb coding, which is certainly something Apple is no stranger to.

Re: Transformer architecture optimized for Apple Silicon

#122

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I would say 1 or 2 years at the OS level

Basically the curtain has been drawn and shows that current LLM’s are just very inefficient and will be optimized in weeks. Whatever improvements Apple’s Neural chip offers just needs more RAM closer to it, which will likely come at the next hardware fresh where this year its probably too late, while whatever is released at the end of 2024 will be good enough.

Re: Transformer architecture optimized for Apple Silicon

#123
post #102

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Live translation on AirPods, here we go!

a babelfish in your ear

Re: Transformer architecture optimized for Apple Silicon

#124
post #113

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

That's highly unlikely. I think most people can't fathom how huge is the scale of GPT-4 and what computing power it requires. Even if 10x optimizations, 10x performance and 10x memory (highly unlikely in 5 years) that's not going to be sufficient to run it locally.

I think people are overestimating how many parameters GPT-4 is really trained on.

While also overestimating how many parameters anybody really needs after fine tuning.

The market will find the sweet spot. Right now everyone’s tinkering with the 7B parameter LLM and then going to move up to the 65B one once they've refined the process. I think its fiction that anybody really needs a 10 trillion parameter LLM at all. It will be completely niche.

Re: Transformer architecture optimized for Apple Silicon

#125
post #120
post #39

Earlier quoted context omitted.

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

Hundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds. It's also likely that memory requirements will be shrunk through software and ML improvements.

I don’t think RAM on phones has been increasing exponentially? For good reason — it uses a lot of power.

Re: Transformer architecture optimized for Apple Silicon

#126

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

Same experience. Alexa is DIMENSIONS better than Siri. Siri can create a timer ... okay even an alarm. That’s it. It is comically bad. Their text to speech is excellent, but the rest is unbelievable bad. If Apple has some kind of silver bullet, it’s time to put it out or be left behind.

I don’t have all the details because NDAs but I ask questions and get nods and grins from colleagues and friends at chip companies; Apple (and nVidia and Intels) silver bullet to cloud hosted, software based AI is AI chips.

We’re circling back around to local compute being the default as hardware performance of next gen phones and tablets reaches a “good enough” point for most users.

There will be scientific problems that will require modern server clusters but most consumer facing AI needs will be done on hardware within a decade.

I’m not saying AGI in a decade, I’m saying cutting edge logic embedded in software now will be the basis for logic in chips in the years to come.

Re: Transformer architecture optimized for Apple Silicon

#127

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

[deleted]

Re: Transformer architecture optimized for Apple Silicon

#128
post #114

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn…

Jobs has been dead for over ten years

Re: Transformer architecture optimized for Apple Silicon

#129
post #111

Earlier quoted context omitted.

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as the…

Offtopic, but for what purpose are you running llms locally (especially everyday)? My understanding was that the prompting requires to make them work at all was too great.

A little bit of research, a little bit of actual useful tasks - I'm interested in summarisation, which alpaca is decent at (even compared to existing summarisation-specific models I've tried)

My other motivation is making sure I understand what offline LLMs can do... while I use GPT-3 and 4 extensively, I don't want to send something over the wire if I don't have to (e.g. if I can summarise e-mails locally, I'd rather do that than send them to OpenAI).

It's also surprisingly good at defining things if I'm somewhere with no internet connectivity and want to look something up (although obviously that's not really what it's good at & hallucination risks abound)

Re: Transformer architecture optimized for Apple Silicon

#130

Earlier quoted context omitted.

They do have a search engine though with Siri Suggestions. Which has a user base of ~1.5b+ people. We only see the tip of the iceberg so who knows what else it is capable of.

“Hey siri, what is today’s date?” “Sorry, I’m having trouble connecting to the network”

Me: Hey Siri, call Toyota.

Siri: I'm sorry, that contact is not in your list

Me: Siri, what is the number for Toyota.

Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go]

Me: Siri, call Toyota

Siri: I'm sorry, you do not have that contact number.

Me: &$@@&/&&/&!!!

Post reply on HN