Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

301–310 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#301

Post by cal85 »

parker_mountain wrote: post #88
arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

> apple has shipped everything required for you to point at a light /in your house/ to turn it on or off

Point at it? You mean with a finger? What do you mean they've shipped everything required?

Re: Transformer architecture optimized for Apple Silicon

#302

Post by danieldrehmer »

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Why 5 years?

I'm running a totally usable 13b parameters llama model in my macbook air, which seems to give outputs equivalent to what I was getting from GPT3 in June 2022.

How much more hardware would it really be needed for GPT-4 level outputs natively? Perhaps software optimizations alone could do most of the trick.

Re: Transformer architecture optimized for Apple Silicon

#303

Post by sebzim4500 »

JCharante wrote: post #268
simonh wrote: post #15

Earlier quoted context omitted.

It this provides optimised hardware acceleration on Apple Silicon for low level machine learning libraries. PyTorch is supported for example, it’s a machine learning library with GPU acceleration that’s been around for 6 years now. It’s used in a few commercial projects, including Tesla Autopilot. It can be used for natural language processing, image manipulation, and possibly to build an LLM I suppose, but as a low…

My understanding is that llama.cpp is a port of meta’s llama release that uses PyTorch. Then I hope we can run llama with Apple Neural Engine.

llama.cpp does not use PyTorch, it uses a niche ML library made by its creator that only runs on the CPU.

Re: Transformer architecture optimized for Apple Silicon

#304

Post by garblegarble »

_rs wrote: post #194
garblegarble wrote: post #176

Earlier quoted context omitted.

I've been pretty happy with 32GB, but the 30B models do push near to the limits. I don't see a big difference between the quality of 65B (running on a 64GB x86 host) and 30B on M1 (although that may be the 4bit quantisation though, so take that with a grain of salt). I'm just glad that I have it on an M1... I have a 3080 in my PC, but when I got that I was thinking more of Stable Diffusion and YOLO tasks rather than…

Yeah, it is expensive. My other strong consideration is battery life, since DRAM is always running; going from 32 to 64 would be a hit to battery life regardless of workload, but hard to say exactly how big of a hit. I'm curious, which configuration of the M1 MBP do you have?

I went for the 16" with M1 Max w/32 GPU cores and 1TB SSD (500GB free, I offload most large files my NAS/iCloud). On the added power usage, my understanding is that's less of a concern due to using LPDDR5?

The only drawback I've found with the M1 Max model is the added weight from the bigger heatsink just makes it a hair heavier than I'd like when picking it up at the front with one hand when open... and that in the winter time the case is cold no matter what you're running, I used to love that my Intel MBP acted as a mini leg warmer :-)

Re: Transformer architecture optimized for Apple Silicon

#305

Post by [deleted] »

calvinmorrison wrote: post #166
ThorsBane wrote: post #137

Earlier quoted context omitted.

This will be the AI killer app for whoever pulls it off first. GPT-4 or higher performance on a device as nice as an iPhone...will create so many new capabilities for smart phones. It's going to be an amazing time for tech in the near future and beyond.

why does this matter when we have cloud computing and high speed mobile internet? why does it need to be local

[deleted]

Re: Transformer architecture optimized for Apple Silicon

#306

Post by garblegarble »

johnthuss wrote: post #182
garblegarble wrote: post #129

Earlier quoted context omitted.

A little bit of research, a little bit of actual useful tasks - I'm interested in summarisation, which alpaca is decent at (even compared to existing summarisation-specific models I've tried) My other motivation is making sure I understand what offline LLMs can do... while I use GPT-3 and 4 extensively, I don't want to send something over the wire if I don't have to (e.g. if I can summarise e-mails locally, I'd rathe…

What prompt are you using for summarization? I’ve tried several variations without consistent results.

On alpaca, I've found "Below is an instruction that describes a task. Write a response that appropriately completes the request. Summarise the following text: " or "Give me a 5 word summary of the following: " to work fairly well using the 30B weights.

It's certainly nowhere close to the quality of OpenAI summarisation, just better than what I previously had locally (e.g. in summarising a family history project with transcripts of old letters, gpt-3.5-turbo was able to accurately read between the lines summarising an original poem which I found amazing).

I half wonder if the change in spelling from US -> UK makes a difference...

I'd run a test on that but I've just broken my alpaca setup for longer prompts (switched to use mainline llama.cpp, which required a model conversion & some code changes, and it's no longer allocating enough memory)

Re: Transformer architecture optimized for Apple Silicon

#307

Post by agentcoops »

arthurcolle wrote: post #84
davnicwil wrote: post #19

Earlier quoted context omitted.

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

Honestly, it speaks mountains to Apple as an organisation that they did not invest all of their resources into Siri, especially when you consider how Amazon invested in Alexa etc. Everyone has long understood that the "personal assistant" represents really the pinnacle of not just what AI but the personal computer should be able to offer -- the mistake this past decade was thinking that had to do with then current ML paradigms or with these completely useless voice agents. That Apple could realise it was a dead-end and not sink much more investment/engineering resources into the tool any further than what anyone actually uses these tools for (changing song, lowering volume, etc) is outstanding.

Ultimately, I'm very reassured that it's Apple and Microsoft leading the way for personal and business AI assistants respectively. As Stratechery has emphasised, the latter not only already has all the data from most firms outside Silicon Valley, but more importantly has always adhered to and designed their software according to a computer-as-a-tool philosophy versus Google's "ML/AI will do everything for you behind the scenes"; the former has not only made a show of data security but invested massively in the hardware architectures necessary to making on-device AI a possibility. Without at all being a "fanboy", Apple is quite literally the only company I would trust my personal data to for use in a GPT-based assistant.

Re: Transformer architecture optimized for Apple Silicon

#308

Post by wwalexander »

FredPret wrote: post #251
post-it wrote: post #221

Earlier quoted context omitted.

It's probably not glasses. Nobody likes wearing glasses. Every attempt at making tech glasses has failed. It'll probably be the Apple Watch.

Everybody would love wearing Apple Glasses. Would probably look really cool. I remember seeing AirPods in somebody's ears for the first time. "What a dork" I thought back then. Now they're cool and everywhere.

I would not love wearing them. I waste a significant amount of money on contact lenses each year for the privilege of not having to wear anything on my face.

Re: Transformer architecture optimized for Apple Silicon

#309

Post by sheepscreek »

singularity2001 wrote: post #231
sheepscreek wrote: post #9

Earlier quoted context omitted.

#2. A way to optimize running LLMs locally on Apple Silicon (including iPhones) I am just a little better informed. As I understand it, their code improves model performance and memory consumption using PyTorch and Huggingface libraries.

it's python code so not for iphone (?)

Sorry - I should have been more clear.

Their examples compared the A* CPU performance and the repo includes Swift only code samples. But they’ve also made it possible to use them with traditional tooling (torch, huggingface).

Hope that helps explain it.

Re: Transformer architecture optimized for Apple Silicon

#310

Post by UncleEntity »

highwaylights wrote: post #257
lotsofpulp wrote: post #179

Earlier quoted context omitted.

I only use Siri with the original Homepods. It has always been terrible. You cannot even tell it to start playing movies on the TV that you have already purchased from Apple itself in the TV app. It will start to play some random song or something. I should be able to say hey siri, start playing on , and it should be able to start the TV and start playing it.

A friend has one that constantly plays U2 out of nowhere. He doesn't even listen to U2, and maintains that he's never requested it. This isn't in response to a prompt either. He'll unpause the connected TV and it'll just start blasting U2.

Yes, the U2 Songs of Innocence album.

In case someone doesn’t know they gave that album to all iTunes users as part of a promotional thing when it came out.

On my iPhone if I connect the Bluetooth headset and accidentally push the call/cancel button out comes U2…

Post reply on HN