Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

191–200 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#191

Post by smoldesu »

king_magic wrote: post #171
015a wrote: post #121

Earlier quoted context omitted.

If Siri can't give you the time without the internet, I think you need to update iOS. There are definitely two tiers of Siri queries. There are queries like "set the brightness to 10%" or "set a timer for 5 minutes" which absolutely and consistently work without internet, and have for several years, and if you're legitimately having a different experience then its possible a cosmic ray hit your iPhone (or, realistica…

> If Siri can't give you the time without the internet, I think you need to update iOS. interesting - latest iOS in airplane mode - "hey siri what time is it?" - "you need to turn off airplane mode to do that" but - "hey siri set a time for 5 minutes" - works just fine

They must be using the esteemed time delivery API, https://www.whattimeisitrightnow.com/

Re: Transformer architecture optimized for Apple Silicon

#192

Post by lunatuna »

wetpaws wrote: post #187
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

Apple's exceptional hardware can only be rivaled by the absolute shitfest quality of their software. So no, I'm not buying it.

Exactly. You can see Apple struggle to build anything of quality with their software stack. Good luck making anything from this. So much great HW sadly wasted.

Re: Transformer architecture optimized for Apple Silicon

#194

Post by _rs »

garblegarble wrote: post #176
_rs wrote: post #155

Earlier quoted context omitted.

Off topic slightly, but are you running into limits with 32GB RAM that the 64GB model would meaningfully be adequate for? Do you wish you had one of the larger RAM models?

I've been pretty happy with 32GB, but the 30B models do push near to the limits. I don't see a big difference between the quality of 65B (running on a 64GB x86 host) and 30B on M1 (although that may be the 4bit quantisation though, so take that with a grain of salt). I'm just glad that I have it on an M1... I have a 3080 in my PC, but when I got that I was thinking more of Stable Diffusion and YOLO tasks rather than…

Yeah, it is expensive. My other strong consideration is battery life, since DRAM is always running; going from 32 to 64 would be a hit to battery life regardless of workload, but hard to say exactly how big of a hit.

I'm curious, which configuration of the M1 MBP do you have?

Re: Transformer architecture optimized for Apple Silicon

#195

Post by bloudermilk »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

I would gladly install a 1GB+ app on my 512GB iPhone that never gets above 50GB because of iCloud’s remote storage optimizations.

Re: Transformer architecture optimized for Apple Silicon

#196

Post by vbezhenar »

Tiktaalik wrote: post #184
BonoboIO wrote: post #107

Earlier quoted context omitted.

Same experience. Alexa is DIMENSIONS better than Siri. Siri can create a timer ... okay even an alarm. That’s it. It is comically bad. Their text to speech is excellent, but the rest is unbelievable bad. If Apple has some kind of silver bullet, it’s time to put it out or be left behind.

Amazon has publicly walked back from Alexa because turns out people don't use it for much beyond creating a timer and it doesn't make money. I think it's quite likely that Apple realized the same quite a bit earlier and eased up on Siri versus focusing on other things.

Honestly I observe zero progress in the last 10 years. My first iPhone was 4S. My current iPhone is 8. My use-case is absolutely the same. Calls, SMS, Whatsapp, navigation, books, browser. The same with macs. I boot, open IDE, Terminal and that's about it. Yes, hardware's getting faster (which is compensated by software getting slower). But nothing changed dramatically.

I think that the only thing that changed is adoption of fingerprint reader. Both in iPhone 8 and in mac.

AI might be that thing that could tremendously change my usage pattern. But it should be as smart as human assistant. GPT4-level intelligence might have the necessary power for that.

Re: Transformer architecture optimized for Apple Silicon

#197

Post by ronyfadel »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

That is not even taking into consideration that shipping your weights to the client is akin to giving your product away.

Re: Transformer architecture optimized for Apple Silicon

#198

Post by gumby »

sidibe wrote: post #180
gumby wrote: post #114

Earlier quoted context omitted.

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn…

> He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple I'm sure he's great but this made me laugh

I aim to please!

(And yes he’s a great guy)

Re: Transformer architecture optimized for Apple Silicon

#199

Post by ioedward »

dev_tty01 wrote: post #73
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I'm expecting the Mac Pro to have AI/ML massive GPU modules and/or specialized ML dedicated compute available as add-ons for in-house training. For a not-so-small fee of course. They have all the pieces, it just needs to be brought to bear on the ML training market.

No it won't. Large language models are trained on 1,000 - 50,000 GPUs. No one's going to buy hundreds of Mac pros to mount them in a datacenter for training ML models.

Re: Transformer architecture optimized for Apple Silicon

#200

Post by LesZedCB »

qumpis wrote: post #112
monocasa wrote: post #66

Earlier quoted context omitted.

The rumor I've heard is that GPT4 didn't meaningfully increase the parameter count versus GPT3.5, but instead focused on training and structural improvements.

Well the inference time of gpt4 seems to be far greater than gpt3, so it could hint a difference in parameters count.

if you watch their announcement Livestream video it looked just as fast as normal ChatGPT.

I think what we have access to is a fair bit slower.

Post reply on HN