Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

171–180 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#171

Post by king_magic »

015a wrote: post #121
belugacat wrote: post #63

Earlier quoted context omitted.

Siri is so bad it needs to be scrapped and rethought from the ground up (it can’t even give me the time when internet is down at my rural house). No one at Apple who wants a career there would dare propose that. The switch to a LLM architecture could be the perfect transition point for this.

If Siri can't give you the time without the internet, I think you need to update iOS. There are definitely two tiers of Siri queries. There are queries like "set the brightness to 10%" or "set a timer for 5 minutes" which absolutely and consistently work without internet, and have for several years, and if you're legitimately having a different experience then its possible a cosmic ray hit your iPhone (or, realistica…

> If Siri can't give you the time without the internet, I think you need to update iOS.

interesting - latest iOS in airplane mode - "hey siri what time is it?" - "you need to turn off airplane mode to do that"

but - "hey siri set a time for 5 minutes" - works just fine

Re: Transformer architecture optimized for Apple Silicon

#172

Post by bradDonniger »

kanyethegreat wrote: post #132
bradDonniger wrote: post #54

Earlier quoted context omitted.

They under estimate chip makers plans to embed AI in silicon. Chips intended for launch in 5-6 years are in planning stages right now. Apple, nVidia, and Intel could bring serious hurt to software companies in the next 5-10. Open source purists will weep but really most people do not care, and tech should not merely serve the dedicated.

> Apple, nVidia, and Intel could bring serious hurt to software companies in the next 5-10. What does that mean? What software companies?

Cloud based AI https://news.ycombinator.com/item?id=35283852

Re: Transformer architecture optimized for Apple Silicon

#174

Post by gumby »

duped wrote: post #128
gumby wrote: post #114

Earlier quoted context omitted.

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn…

Jobs has been dead for over ten years

And it feels like nobody has worked on it since then

Re: Transformer architecture optimized for Apple Silicon

#175

Post by gnicholas »

epistasis wrote: post #130
lotsofpulp wrote: post #83

Earlier quoted context omitted.

“Hey siri, what is today’s date?” “Sorry, I’m having trouble connecting to the network”

Me: Hey Siri, call Toyota. Siri: I'm sorry, that contact is not in your list Me: Siri, what is the number for Toyota. Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go] Me: Siri, call Toyota Siri: I'm sorry, you do not have that contact number. Me: &$@@&/&&/&!!!

I work around this by asking it to find Toyota in . Then if it finds the dealership, it asks if I want to navigate there, or call them.

I find the worst errors to be when I ask it for information, or to send a text, and it instead places a phone call to someone. It will even call people that I have never called on my phone (a fact it should know), without asking first.

Re: Transformer architecture optimized for Apple Silicon

#176

Post by garblegarble »

_rs wrote: post #155
garblegarble wrote: post #90

Earlier quoted context omitted.

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as the…

Off topic slightly, but are you running into limits with 32GB RAM that the 64GB model would meaningfully be adequate for? Do you wish you had one of the larger RAM models?

I've been pretty happy with 32GB, but the 30B models do push near to the limits. I don't see a big difference between the quality of 65B (running on a 64GB x86 host) and 30B on M1 (although that may be the 4bit quantisation though, so take that with a grain of salt). I'm just glad that I have it on an M1... I have a 3080 in my PC, but when I got that I was thinking more of Stable Diffusion and YOLO tasks rather than LLMs, and it just doesn't have the VRAM for LLMs.

Alpaca seems like it could be significantly improved with better training (some of the old training data was truncated), so I think there's a decent amount of improvement to be had at the current model size.

In the future though... what would really be a meaningful change would be a larger context size - the 8k tokens of GPT-4 was a big improvement for my uses... I would guess a future local llm with larger context would exceed 32GB, but that's speculation beyond my expertise, I don't know how context size and network size scale.

If it was a PC I'd say go for 64GB, but hard to recommend that given how much Apple charge for RAM upgrades. On my next upgrade (2+ years time, hopefully) I'll likely opt for 64GB+ though

Re: Transformer architecture optimized for Apple Silicon

#177

Post by fingerlocks »

epistasis wrote: post #130
lotsofpulp wrote: post #83

Earlier quoted context omitted.

“Hey siri, what is today’s date?” “Sorry, I’m having trouble connecting to the network”

Me: Hey Siri, call Toyota. Siri: I'm sorry, that contact is not in your list Me: Siri, what is the number for Toyota. Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go] Me: Siri, call Toyota Siri: I'm sorry, you do not have that contact number. Me: &$@@&/&&/&!!!

Just tried this exact interaction. Siri gave me a list of nearest Toyotas and made the call. Seems to be working just fine.

Starting to think all these Siri complaints are either made up or really outdated.

Re: Transformer architecture optimized for Apple Silicon

#178

Post by gcanyon »

epistasis wrote: post #130
lotsofpulp wrote: post #83

Earlier quoted context omitted.

“Hey siri, what is today’s date?” “Sorry, I’m having trouble connecting to the network”

Me: Hey Siri, call Toyota. Siri: I'm sorry, that contact is not in your list Me: Siri, what is the number for Toyota. Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go] Me: Siri, call Toyota Siri: I'm sorry, you do not have that contact number. Me: &$@@&/&&/&!!!

The perfectly obvious use cases Siri overlooks are maddeningly numerous. My go to example was "text this photo to " For years I checked each new iOS release to see if that was enabled, and for close to ten years, no. I'm not sure whether it was the most recent, or the one before (I gave up somewhere along the way) but now you can (finally) do it.

Re: Transformer architecture optimized for Apple Silicon

#179

Post by lotsofpulp »

fingerlocks wrote: post #177
epistasis wrote: post #130

Earlier quoted context omitted.

Me: Hey Siri, call Toyota. Siri: I'm sorry, that contact is not in your list Me: Siri, what is the number for Toyota. Siri: Toyota's phone number is 123-456-7890 [said too fast to remember or write down in one go] Me: Siri, call Toyota Siri: I'm sorry, you do not have that contact number. Me: &$@@&/&&/&!!!

Just tried this exact interaction. Siri gave me a list of nearest Toyotas and made the call. Seems to be working just fine. Starting to think all these Siri complaints are either made up or really outdated.

I only use Siri with the original Homepods. It has always been terrible. You cannot even tell it to start playing movies on the TV that you have already purchased from Apple itself in the TV app. It will start to play some random song or something.

I should be able to say hey siri, start playing on , and it should be able to start the TV and start playing it.

Re: Transformer architecture optimized for Apple Silicon

#180

Post by sidibe »

gumby wrote: post #114
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn…

> He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple

I'm sure he's great but this made me laugh

Post reply on HN