Transformer architecture optimized for Apple Silicon • Hacker News Board

Re: Transformer architecture optimized for Apple Silicon

#151

Post by abraxas » Fri, Mar 24, 2023, 2:03 AM UTC

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Greed is a fabulous motivator but it can't surpass the laws of physics. This is probably why we don't have a cure for cancer or room temperature superconductors or jurassic parks.

It may be so that scaling down a language model like GPT 4 is not possible on hardware systems orders of magnitude smaller than the one used by OpenAI.

I'm not saying it's impossible but it's fallacious to just assume outright that it's inevitable because market forces.

For all we know it could turn out that the only way to get a gpt in a pocket format is through some form of analog chips that must be individually trained that Hinton aptly described as an era of mortal computing.

Re: Transformer architecture optimized for Apple Silicon

#152

Post by levidos » Fri, Mar 24, 2023, 2:06 AM UTC

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Why would they want to give out their trusty LLMs when they can just hide it behind an API, and making sure you'll be dependent on them forever

Re: Transformer architecture optimized for Apple Silicon

#153

Post by parker_mountain » Fri, Mar 24, 2023, 2:09 AM UTC

afavour wrote: post #99

parker_mountain wrote: post #88

Earlier quoted context omitted.

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

“Yes Siri is garbage but it doesn’t matter because Apple are probably working on a top secret new thing that will blow everyone else out of the water” You might be right but there’s no real evidence for it yet.

>because Apple are probably working on a top secret new thing that will blow everyone else out of the water”

that's a mischaracterization of what i said. what i said was that they have been clearly hiring key positions and cannibalizing the siri team for something new. that something new will likely be a major release. i also believe that the ar headset is the unifying product under which they're rallying.

Re: Transformer architecture optimized for Apple Silicon

#154

Post by lr1970 » Fri, Mar 24, 2023, 2:09 AM UTC

The github repo has only 5 commits in it and the last one is August 9, 2022. It looks abandoned.

Re: Transformer architecture optimized for Apple Silicon

#155

Post by _rs » Fri, Mar 24, 2023, 2:10 AM UTC

garblegarble wrote: post #90

s1k3s wrote: post #39

Earlier quoted context omitted.

Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as the…

Off topic slightly, but are you running into limits with 32GB RAM that the 64GB model would meaningfully be adequate for? Do you wish you had one of the larger RAM models?

Re: Transformer architecture optimized for Apple Silicon

#156

Post by parker_mountain » Fri, Mar 24, 2023, 2:11 AM UTC

ngai_aku wrote: post #135

parker_mountain wrote: post #88

Earlier quoted context omitted.

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i…

> apple has shipped everything required for you to point at a light /in your house/ to turn it on or off (and optionally flick). this includes software - the individual components are built and ready - the only thing missing is gluing it together Could you expand on this bit? I’m pretty deep in the Apple ecosystem, but I’m not sure what you’re referencing here

Homepods have U1 chips. They can position airtags within your house. You can put an airtag under, or adjacent to, a lamp and link it to a smart switch. Your apple watch has gesture detection (still in beta, tbf), as well as U1 and real time positioning capabilities.

Everything required for apple to know not only where you are, but which way your hand is pointing, as well as where "smart devices" are in your house is already being sold and rolled out en masse to the majority of people in the ecosystem.

Re: Transformer architecture optimized for Apple Silicon

#157

Post by gitfan86 » Fri, Mar 24, 2023, 2:11 AM UTC

btotes wrote: post #143

gitfan86 wrote: post #134

Earlier quoted context omitted.

NVDA has the potential to hit huge profit growth in the short term, but not long term

What makes you say that?

There may be a short term panic to get position of CUDA GPUs which will allow NVDA to increase their prices a lot. But then other chip makers and software creator will build alternatives to NVDA

Re: Transformer architecture optimized for Apple Silicon

#158

Post by fbdab103 » Fri, Mar 24, 2023, 2:12 AM UTC

BonoboIO wrote: post #107

arthurcolle wrote: post #84

Earlier quoted context omitted.

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give…

Same experience. Alexa is DIMENSIONS better than Siri. Siri can create a timer ... okay even an alarm. That’s it. It is comically bad. Their text to speech is excellent, but the rest is unbelievable bad. If Apple has some kind of silver bullet, it’s time to put it out or be left behind.

Can we take a minute to comment on how bad the text correction is on the iPhone?

Re: Transformer architecture optimized for Apple Silicon

#159

Post by astrange » Fri, Mar 24, 2023, 2:12 AM UTC

lotsofpulp wrote: post #83

threeseed wrote: post #78

Earlier quoted context omitted.

They do have a search engine though with Siri Suggestions. Which has a user base of ~1.5b+ people. We only see the tip of the iceberg so who knows what else it is capable of.

“Hey siri, what is today’s date?” “Sorry, I’m having trouble connecting to the network”

If you're asking for an LLM this will get worse because the first L means "large", as in too large to fit in a phone without a network connection.

Re: Transformer architecture optimized for Apple Silicon

#160

Post by quickthrower2 » Fri, Mar 24, 2023, 2:13 AM UTC

levidos wrote: post #152

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

Why would they want to give out their trusty LLMs when they can just hide it behind an API, and making sure you'll be dependent on them forever

Running locally and “give you” are two very different things. Financially you paid for the phone. From a lock in point of view the chip will be proprietary and very hard to reverse engineer. They can indeed hide it behind a local API that apps that they can ban at will would need to use.