Transformer architecture optimized for Apple Silicon • Hacker News Board

Re: Transformer architecture optimized for Apple Silicon

#331

Post by smoldesu » Fri, Mar 24, 2023, 7:20 PM UTC

brookst wrote: post #329

smoldesu wrote: post #319

Earlier quoted context omitted.

> whether tbey pursue it or not remains to be seen. Pursue what though? UMA is cool, but kinda meaningless if the majority of Macbooks are min-spec. That leaves you with 4-5gb of VRAM, assuming you've left nothing open. What is Apple going to do with that UMA that other manufacturers cannot? It's certainly nice that 128gb Macs exist for models that might be too big to otherwise load into memory. It's useless for prod…

> Pursue what though? A Mac variant that trades CPU cores for GPU/ML cores while having 192GB+ of UMA memory. > I struggle to imagine the “opportunities” Two of them: 1) academic / R&D compute, where people could have at least A6000 class GPU on the desktop, and 2) cloud inference servers, probably for Apple’s own services. I’m not saying they will or should do those things, just that the apple silicon arch is well p…

Academics might buy in, but they're a small market and still fairly easy to poach with quality server hardware. You may be right about Apple using them for cloud inferencing though, seeing how they'd rather pound sand than patch up their relationship with Nvidia.

Whichever way you look at it though, neither of those are really opportunities. Apple boxed themselves into the consumer market, and now has to compete with professionally-priced products.

Re: Transformer architecture optimized for Apple Silicon

#332

Post by gcanyon » Fri, Mar 24, 2023, 7:36 PM UTC

Tagbert wrote: post #324

gcanyon wrote: post #178

Earlier quoted context omitted.

The perfectly obvious use cases Siri overlooks are maddeningly numerous. My go to example was "text this photo to " For years I checked each new iOS release to see if that was enabled, and for close to ten years, no. I'm not sure whether it was the most recent, or the one before (I gave up somewhere along the way) but now you can (finally) do it.

"HeySiri, send this photo to name in messages." "Ready to send" "Send it" "It's sent" done Perhaps the verb "text" is unclear to Siri?

Like I said: it does it now. It didn't from inception until something like 2021.

Re: Transformer architecture optimized for Apple Silicon

#333

Post by sedatk » Fri, Mar 24, 2023, 9:52 PM UTC

brookst wrote: post #312

sedatk wrote: post #64

Earlier quoted context omitted.

Local CSAM scanning wasn't an open book either. Also, it wasn't mutually exclusive with cloud scanning.

It was pretty open — auditable and cryptographically proven sources for all triggering hashes, auditable updates. And it was mutually exclusive with cloud scanning, as it was part of E2EE.

About mutual exclusivity, you may be right. I probably misremember that specific detail, but that was before Apple enabled E2EE on iCloud. It was removed as a feature back then when they suggested local CSAM scanning.

"It was pretty open"

Not really. The scanning code wasn't open source. Apple didn't explain how their proprietary hashing algorithm worked, let alone providing any source code. It was just the promise of Apple about how the whole thing would work. If that's sufficient, then iCloud scanning is as open as local scanning.

Auditability of DB updates doesn't mean much either. Even if a third party organization detected "new additions" to the CSAM DB, what would be the next step? There would be no way to verify if those hashes actually correspond to CSAM. A dictatorship, for example, would just say "yes it's CSAM, trust us", and you'd have no option but to trust their word. Even in the US, there is no way to verify if CSAM DB fully corresponds to actual CSAM. It's just NCMEC's word. We simply don't know.

Re: Transformer architecture optimized for Apple Silicon

#334

Post by apatheticonion » Fri, Mar 24, 2023, 11:10 PM UTC

As much as I love the progress in AI (also see Microsoft's recent Office Copilot) - I seriously think that government needs to step in to regulate the fair use of training data.

Though this is painted by my personal beliefs, in order to maximise innovation, I believe it's the role of government to implement regulations that support a competitive commercial environment. Without this, monopolies will form around hard to obtain resources, innovation will stagnate and consumers will be subject to exploitation.

Currently, data is mostly acquired without user consent and is accessible retroactively. Companies own that data, they can trade it and they can use it however they want to. You as a consumer have no say in this and it's virtually impossible to live a normal life without being the subject of analysis.

While it's incredible that companies can produce undeniably valuable products like Copilot - ultimately - they will profit from these products. The irony is they built them from data sourced from you, likely from something you paid for (MS Word, etc).

The key ingredient in these products is training data. If you wanted to compete with them, no matter how capable you are as an engineer, you could never make Copilot without the same scale of data Microsoft has gathered.

I don't know what kind of regulation would even out the playing field, but I wouldn't mind being compensated for my role in creating these highly profitable products.

Re: Transformer architecture optimized for Apple Silicon

#335

Post by brookst » Sat, Mar 25, 2023, 2:43 AM UTC

smoldesu wrote: post #331

brookst wrote: post #329

Earlier quoted context omitted.

> Pursue what though? A Mac variant that trades CPU cores for GPU/ML cores while having 192GB+ of UMA memory. > I struggle to imagine the “opportunities” Two of them: 1) academic / R&D compute, where people could have at least A6000 class GPU on the desktop, and 2) cloud inference servers, probably for Apple’s own services. I’m not saying they will or should do those things, just that the apple silicon arch is well p…

Academics might buy in, but they're a small market and still fairly easy to poach with quality server hardware. You may be right about Apple using them for cloud inferencing though, seeing how they'd rather pound sand than patch up their relationship with Nvidia. Whichever way you look at it though, neither of those are really opportunities. Apple boxed themselves into the consumer market, and now has to compete with…

I think you vastly overestimate the emotionality of corporate execs.

Re: Transformer architecture optimized for Apple Silicon

#336

Post by waffletower » Sat, Mar 25, 2023, 6:46 AM UTC

davnicwil wrote: post #19

endisneigh wrote: post #7

i'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that opena…

I really think you have hit the nail on the head here. Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. We must be talking orders of magnitude differences in operational cost, not to mention completely unique features like privacy. The very definition of disruption, waitin…

iCloud has screamed ineptitude for decades. I don't see how Apple can possibly develop successful products that can't be held in the hand and fetishized by myopic designers. Much like a stereotypical grandparent raised during the Great Depression, Tim Cook still leads like a supply chain miser trying to save a company that has long prospered -- they tragically squander the unprecedented capital at their disposal. Change my mind, Apple please.

Re: Transformer architecture optimized for Apple Silicon

#337

Post by smugma » Sat, Mar 25, 2023, 5:13 PM UTC

yurishimo wrote: post #270

highwaylights wrote: post #255

Earlier quoted context omitted.

> Now they're cool and everywhere. One of these things is true.

There is a difference between the subjective and objective "cool." They've sold something like 150 million pairs of Airpods? I think that qualifies as objectively "cool."

If lots of people are doing or buying something, that’s a signal that it isn’t cool.

Re: Transformer architecture optimized for Apple Silicon

#338

Post by smugma » Sat, Mar 25, 2023, 5:35 PM UTC

endisneigh wrote: post #36

brookst wrote: post #33

Earlier quoted context omitted.

What other services does Apple charge a subscription fee for despite having no opex?

apple fitness I imagine has a pretty small opex (sure they pay trainers, but that doesn't scale linearly with subscriptions. the videos themselves are limited in scope and could be cached pretty trivially. the actual fitness tracking happens on device), same with apple arcade which I highly doubt has any additional expensive over the app store in general. I can see it - "Siri+", pay $5 a month for fine tuned model up…

If you look at Peloton’s financial statements, you’ll see that your assumptions are off. For one thing, music costs are significant (Peloton had to settle a $200M law suit for not doing it right).

In addition to the trainers, you have producers, graphic designers, artist spotlights, celebrity interviews, etc.

Yes, some costs are fixed so with more subs you get higher profitability but that’s really different than charging for something that doesn’t have a variable cost.

Re: Transformer architecture optimized for Apple Silicon

#339

Post by 015a » Sat, Mar 25, 2023, 7:35 PM UTC

hobs wrote: post #189

015a wrote: post #103

Earlier quoted context omitted.

They have every query their users enter into Spotlight. They have all the results it returns (who cares if they originated from Google, ok maybe they do, maybe not using it for anything is part of their deal with Google). They have your contacts, calendars, messages, email, music, workout history, tasks, real time location, which Siri already uses to e.g. recommend a destination when you get in your car to a degree o…

Apple does not have the queries that their users enter into spotlight, They dont have all the results it returns. You dont spawn a network request on every local search on an apple device.

Yes you do. It trivially does. Search for "Burger King" and see how many results that generates which couldn't possibly be served locally (undownloaded Apps from the App Store, Maps, search recommendations, etc).

They may not be storing these queries or using them for further analytics, training, etc. But they could.

Re: Transformer architecture optimized for Apple Silicon

#340

Post by 015a » Sat, Mar 25, 2023, 7:41 PM UTC

smoldesu wrote: post #190

015a wrote: post #103

Earlier quoted context omitted.

They have every query their users enter into Spotlight. They have all the results it returns (who cares if they originated from Google, ok maybe they do, maybe not using it for anything is part of their deal with Google). They have your contacts, calendars, messages, email, music, workout history, tasks, real time location, which Siri already uses to e.g. recommend a destination when you get in your car to a degree o…

> And cost effective training and inference silicon is a moat, because its Physical, and like literally One Company on the planet makes it, I got Nvidia on my paper. Did I do the math wrong?

TSMC manufactures the silicon for nearly all of the current generation high end silicon. The only exceptions I've been able to find are, naturally, Intel's CPU lineup (which has nothing to do with AI) (and, to be clear, Intel Arc is also manufactured by TSMC), and the Snapdragon 8cxg3, which is manufactured by Samsung; and the "advanced-ness" of the 8cxg3 is pretty miserable compared to A16, M2, RTX 4000, RDNA3, or Tensor Processing Units.

Apple has purchased the entirety of TSMC's 3nm production for the next 1 to 2 years. They can do that, and can continue to do that, because they have a ton of money. They have a ton of money for buying chips because there isn't some crazy business middle-logic justifying the cost of these chips' performance; they buy chips, they sell chips to consumers. In comparison, literally zero other customers of TSMC derive the majority of their hardware revenue from selling to consumers. Companies like Nvidia make some money this way, but most of their money goes to data center sales, which has their own business justification for buying them which fluctuates (ChatGPT subscriptions? training models? is the past model good enough? etc).