Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's On-Device and Server Foundation Models

machinelearning.apple.com

Post reply on HN
Print view View original

141–150 of 562 posts

Re: Apple's On-Device and Server Foundation Models

#141

Post by xwolfi »

bigyikes wrote: post #11
mdhb wrote: post #5

Earlier quoted context omitted.

So built on stolen data essentially.

Does that imply I just stole your comment by reading it? No snark intended; I’m seriously asking. If the answer is “no” then where do you draw the line?

But then if I write a Pulitzer prize article called "No snark intended: How the web became such a toxic place", where your comment, and all other of ur comments for good measure, figure prominently while I ridicule you and this habit of dumbing down complex problems to reduce them to little witty bites, maybe you'd feel I stole something.

Not something big, not something you can enforce, but you d feel very annoyed Im making good money on something you wrote while you get nothing. I think ?

Re: Apple's On-Device and Server Foundation Models

#142

Post by revscat »

> With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate.

This seems impressive. Is it, really? I don’t know enough about the subject to judge.

Re: Apple's On-Device and Server Foundation Models

#143

Post by danielmarkbruce »

scosman wrote: post #57

“We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks.” This is huuuuge. I don’t see announcement of 3rd party training support yet, but I imagine/hope it’s planned. One of the hard things about local+private ML is I don’t want every app I download to need GBs of weights, and don’t want a delay when I open a new…

this is pretty stock standard lora.

Re: Apple's On-Device and Server Foundation Models

#144

Post by woadwarrior01 »

brcmthrowaway wrote: post #25
urbandw311er wrote: post #18

Earlier quoted context omitted.

Likely they’ll be able to take advantage of the hardware neural engine and be far more power efficient. Apple has demonstrated this is something it takes pretty seriously.

So iOS LLM Apps dont use the neural engine? Lol

None of the current iOS and macOS LLM Apps use the Neural Engine. They use the CPU and the GPU.

nb: I'm the author of a fairly popular app in that category.

Re: Apple's On-Device and Server Foundation Models

#145

Post by bastawhiz »

revscat wrote: post #142

> With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate. This seems impressive. Is it, really? I don’t know enough about the subject t…

For a phone running locally, that's pretty fast. The bigger question is how good the output is. Fast garbage isn't useful, so we'll have to wait to see what it actually ends up looking like outside of demos.

Re: Apple's On-Device and Server Foundation Models

#146

Post by lossolo »

scosman wrote: post #57

“We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks.” This is huuuuge. I don’t see announcement of 3rd party training support yet, but I imagine/hope it’s planned. One of the hard things about local+private ML is I don’t want every app I download to need GBs of weights, and don’t want a delay when I open a new…

It's LORA, most of the things you saw in Apple Intelligence on device presentation are basically different LORAs.

Re: Apple's On-Device and Server Foundation Models

#147

Post by talldayo »

ksec wrote: post #124

I hope, this could mean Apple will push the baseline of ALL Macs to have higher than 8GB of Memory. While I wish we all get 16GB M4 as baseline. Apple being Apple may only give us 12GB, and charges extra $100 for the 16GB option. It will still be a lot better than 8GB though.

The Steam Deck ships with 16 gigs of quad-channel LPDDR5 and it costs $400. Apple knows exaaaactly what they're doing with this sort of pricing.

Can't forget about that cozy 256gb SSD either. An AI computer will need more than that, right?

Re: Apple's On-Device and Server Foundation Models

#148

Post by idiotsecant »

danielmarkbruce wrote: post #143
scosman wrote: post #57

“We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks.” This is huuuuge. I don’t see announcement of 3rd party training support yet, but I imagine/hope it’s planned. One of the hard things about local+private ML is I don’t want every app I download to need GBs of weights, and don’t want a delay when I open a new…

this is pretty stock standard lora.

[flagged]

Re: Apple's On-Device and Server Foundation Models

#149

Post by tmpz22 »

vzaliva wrote: post #67

I love that they use machinelearning.apple.com not ai.apple.com

For the majority of the keynote they explicitly avoided the word AI instead substituting the word Intelligence, then Apple Intelligence, and then towards the end they said AI and ChatGPT once or twice.

I think they saw the response to all the AI shoveling and Microsoft Recall and executed a fantastic strategy to reposition themselves in industry discussions. I still have tons of reservations about privacy and what this will all look like in a few years, but you really have to take your hat off to them. WWDC has been awesome and it makes me excited to develop for their platform in a way I haven't felt in a very, very, long time.

Post reply on HN