Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformer architecture optimized for Apple Silicon

github.com

Post reply on HN
Print view View original

241–250 of 342 posts

Re: Transformer architecture optimized for Apple Silicon

#241

Post by dragonwriter »

levidos wrote: post #152
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Why would they want to give out their trusty LLMs when they can just hide it behind an API, and making sure you'll be dependent on them forever

> Why would they want to give out their trusty LLMs

OpenAI won’t want to; open source competitors already are, and they will keep getting better. The more of a lead OpenAI has over commercial competitors, the more incentive those commercial competitors will have to back open source options.

Re: Transformer architecture optimized for Apple Silicon

#242

Post by KennyBlanken »

ronyfadel wrote: post #193

The bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model ( 1GB.

A number of games are in the 60-150GB range...

Re: Transformer architecture optimized for Apple Silicon

#243

Post by dragonwriter »

Gigachad wrote: post #27
ll_mama wrote: post #18

Earlier quoted context omitted.

5 years? Bit of a long stretch with how much focus is going into AI. I’d be surprised if it’s not possible within two iterations of the new iPhone.

GPT-3 was said to require something like 150gb of VRAM. I don't see that gap being bridged in phones within 2 years.

> GPT-3 was said to require something like 150gb of VRAM.

By all accounts GPT-3 is wildly inefficient in resource use; OpenAI runs like a company that’s concerned with the functionality it can achieve by calendar date, and has an almost infinite bankroll to do it. But, other actors in the field have different priorities, and the various open source or, at least, available-to-use models seem to be far more efficient than the OpenAI models of similar function (though they are behind the newest OpenAI models in function.)

Re: Transformer architecture optimized for Apple Silicon

#244

Post by kergonath »

alwayslikethis wrote: post #161
abraxas wrote: post #151

Earlier quoted context omitted.

Greed is a fabulous motivator but it can't surpass the laws of physics. This is probably why we don't have a cure for cancer or room temperature superconductors or jurassic parks. It may be so that scaling down a language model like GPT 4 is not possible on hardware systems orders of magnitude smaller than the one used by OpenAI. I'm not saying it's impossible but it's fallacious to just assume outright that it's ine…

There is nothing impossible about a Jurassic park in terms of laws of physics though. It's a matter of getting the right materials to recreate dinosaurs. Same goes for cancer cures. I'm not sure if there are physical limitations preventing a room temp superconductor.

> I'm not sure if there are physical limitations preventing a room temp superconductor.

Not really. We just know that it’s very uncommon at best. The limit is material dependent and we already have superconductors with more than one order of magnitude difference in their critical temperature (e.g. ~4 K vs ~40 K; YBCO, which is widely studied, is at 90 K). It is not inconceivable that we could come up with some fancy material with a critical temperature three times as high again. There are several laboratories with good money working on it.

Re: Transformer architecture optimized for Apple Silicon

#245

Post by ohgodplsno »

parker_mountain wrote: post #156
ngai_aku wrote: post #135

Earlier quoted context omitted.

> apple has shipped everything required for you to point at a light /in your house/ to turn it on or off (and optionally flick). this includes software - the individual components are built and ready - the only thing missing is gluing it together Could you expand on this bit? I’m pretty deep in the Apple ecosystem, but I’m not sure what you’re referencing here

Homepods have U1 chips. They can position airtags within your house. You can put an airtag under, or adjacent to, a lamp and link it to a smart switch. Your apple watch has gesture detection (still in beta, tbf), as well as U1 and real time positioning capabilities. Everything required for apple to know not only where you are, but which way your hand is pointing, as well as where "smart devices" are in your house is…

$25 for an Airtag, plus say $20 for a smart switch. Plus $200 for a HomePod.

To turn a single lamp on/off by pointing at it, something that nobody ever wants to do (alright, once for the cool factor). If someone wanted to overpay for useless features, they can already go for a Philips Hue.

I can turn off my lamps from anywhere in the world using a $10 Tuta ZigBee bridge and a $8 LIDL light.

Re: Transformer architecture optimized for Apple Silicon

#246

Post by leodriesch »

pfoof wrote: post #237

So, is Stable Diffusion working finally on TPU or not? DiffusionBee uses GPU and running this https://github.com/apple/ml-stable-diffusion with CPU_AND_NE just segfaults

Amazing AI by Sindre Sorhus uses Apples port of Stable Diffusion to the neural engine.

https://sindresorhus.com/amazing-ai

Re: Transformer architecture optimized for Apple Silicon

#247

Post by kergonath »

atleastoptimal wrote: post #210
endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Local inference is a pipe dream. There's no way any of these companies will allow consumers to interact with their flagship models on anything but a black box api.

The black box could well be a local library. Functionally, what would be the difference between using something like AIKit and using a REST API over the network?

Local has a lot of advantages as well (latency, privacy, etc).

Re: Transformer architecture optimized for Apple Silicon

#249

Post by stn_za »

gitfan86 wrote: post #157
btotes wrote: post #143

Earlier quoted context omitted.

What makes you say that?

There may be a short term panic to get position of CUDA GPUs which will allow NVDA to increase their prices a lot. But then other chip makers and software creator will build alternatives to NVDA

And be so far behind it's not even funny.

Look at AMD vs NVIDIA RTX implementations. Nah, NVDA is a solid buy for long term. They also almost acquired ARM (Which shows where their heads are at at the very least.)

Owning ARM, the thing that is inside ya know, everyone's phones, routers, etc

Re: Transformer architecture optimized for Apple Silicon

#250

Post by ohgodplsno »

brookst wrote: post #100
dev_tty01 wrote: post #73

Earlier quoted context omitted.

I'm expecting the Mac Pro to have AI/ML massive GPU modules and/or specialized ML dedicated compute available as add-ons for in-house training. For a not-so-small fee of course. They have all the pieces, it just needs to be brought to bear on the ML training market.

UMA is the secret. A 256GB Mac Pro can dedicate almost all of that to AI. Their GPUs are the weak spot. The Nvidia 4080 is 2.5x faster than the M2, and the A100 is 15 times faster.

The 4080 is _25x_ faster than the M2 on pure fp32 (which is what most GPUs are doing most of the time). Apple compared the M2 to the laptop 4080, using numbers heavily biased to them (running a 4080 at 10W does tend to make it not perform, yes).

Not a single benchmark in the world has supported Apple's claim that the GPU in the M2 is that powerful. It's just yet another cute embedded GPU that does the job, but nothing more. It's made to push out 8K frames really fast, which it does because of UMA, but want demanding task will have it be eaten alive by any real GPU.

Post reply on HN