Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

261–270 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#261

Post by moonchrome »

post-it wrote: post #221
parker_mountain wrote: post #88

Earlier quoted context omitted.

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

It's probably not glasses. Nobody likes wearing glasses. Every attempt at making tech glasses has failed. It'll probably be the Apple Watch.

I'm not a VR/AR fan but if Apple can come out with super high quality AR headset that can augment/replace monitors I would buy one instantly. I might be earlier than mainstream adopter but if the tech ever gets there this will be a natural shift.

I won't buy Meta products but I keep waiting for someone to do VR workspace right - where I can truly work and travel from anywhere with a decent chair and a desk for keyboard (no need to lug around monitors, no need for huge table).

Re: Transformer architecture optimized for Apple Silicon

#262

Post by bigfudge »

qumpis wrote: post #111
garblegarble wrote: post #90

Earlier quoted context omitted.

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as the…

Offtopic, but for what purpose are you running llms locally (especially everyday)? My understanding was that the prompting requires to make them work at all was too great.

Necessary if you have sensitive datasets you can’t share with US company

Re: Transformer architecture optimized for Apple Silicon

#265

Post by JJJollyjim »

singularity2001 wrote: post #230
hahnchen wrote: post #16

you should put [June 2022] in the title

also no updates since then. does any library follow this reference implementation? llama.cpp surely is more optimized?

llama.cpp runs on the CPU, not the ANE or GPU.

Re: Transformer architecture optimized for Apple Silicon

#266

Post by ithkuil »

jocaal wrote: post #235
feifan wrote: post #87

Earlier quoted context omitted.

The part that Siri is bad at will be commoditized — someone will open-source a GPT-4-level language model. And Apple's moat will be being able to run that on-device with all the attendant benefits (privacy, zero marginal cost to the company, availability in more scenarios, etc)

You realize that these language models are like 100's of GBs in size and consumes 10's of GB's of memory. Last time I checked, apple still ships their products with less than the market average in both of these specs. If you want a local running LLM on an iphone, get ready to sell a kidney.

(assuming we're not talking about the near future)

I think this can be a scenario of converging incentives: on one side large models will incentivized hardware manufacturers to increase the memory available on the devices, while on the other sides model developers will be incentivized to trim the fat on the models and devise compression mechanisms that don't compromise quality too much.

It's not unthinkable to imagine a hand held device able to run full inference locally a few device generations in the future.

Re: Transformer architecture optimized for Apple Silicon

#267

Post by mikkelam »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

Apple is artifically shipping phones with low storage sizes to upsell phones with more space. If there was a big economical advantage for them to have larger models on the phone I expect it would be easy to solve.

Re: Transformer architecture optimized for Apple Silicon

#268

Post by JCharante »

simonh wrote: post #15
hannofcart wrote: post #3

As someone entirely at sea with the rapid pace of development in this sphere: 1. Is this a new LLM from Apple? 2. Is this a way to optimize running LLMs like Llama locally on M1 macs? 3. Something else altogether?

It this provides optimised hardware acceleration on Apple Silicon for low level machine learning libraries. PyTorch is supported for example, it’s a machine learning library with GPU acceleration that’s been around for 6 years now. It’s used in a few commercial projects, including Tesla Autopilot. It can be used for natural language processing, image manipulation, and possibly to build an LLM I suppose, but as a low…

My understanding is that llama.cpp is a port of meta’s llama release that uses PyTorch. Then I hope we can run llama with Apple Neural Engine.

Re: Transformer architecture optimized for Apple Silicon

#269

Post by ayewo »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

Apple typically solves this with device segmentation.

They can announce an iPhone Pro Ultra model that comes with higher RAM and storage capacity along with a souped up Neural Engine, similar to how they do today with how there are differences between the iPhone and the iPhone Pro screen and camera.

Even better, they could bundle the base model for LLAMA with iOS and ship incremental model updates to those iPhone Ultra users (possibly on a monthly subscription).

Re: Transformer architecture optimized for Apple Silicon

#270

Post by yurishimo »

highwaylights wrote: post #255
FredPret wrote: post #251

Earlier quoted context omitted.

Everybody would love wearing Apple Glasses. Would probably look really cool. I remember seeing AirPods in somebody's ears for the first time. "What a dork" I thought back then. Now they're cool and everywhere.

> Now they're cool and everywhere. One of these things is true.

There is a difference between the subjective and objective "cool." They've sold something like 150 million pairs of Airpods? I think that qualifies as objectively "cool."
Post reply on HN