Live data from Hacker News

Apple's On-Device and Server Foundation Models

machinelearning.apple.com

211–220 of 562 posts

Re: Apple's On-Device and Server Foundation Models

#211
post #208
post #204

Earlier quoted context omitted.

I don't think the LoRAs are fine-tuned locally at all. It sounds like they use RAG to access data.

Consider a feature from earlier in the keynote: the thing Notes (and Math Notes) does now where it fixes up your handwriting into a facsimile of your handwriting, with the resulting letters then acting semantically as text (snapping to a baseline grid; being reflowable; being interpretable as math equations) but still having the kind of long-distance context-dependent variations that can't be accomplished by just gen…

For everything other than handwriting I don't think the LoRAs are fine-tuned locally.

Re: Apple's On-Device and Server Foundation Models

#212

It would be interesting to see how these models impact battery life. I’ve tried a few local LLMs on my iPhone 15 Pro via the PrivateLLM app, and the battery charge plummets just after a few minutes of usage.

It looks like PrivateLLM uses the GPU for inferencing, from what I can tell, Apple is using the ANE on the A17 Pro. For M1 and above, I'd presume they are using the GPU since the ANE in M series isn't great.

Re: Apple's On-Device and Server Foundation Models

#214
post #6

Will these smaller on device models lead to a crash in GPU prices?

Not in the short-to-medium-term. Try the local models out, they fall over pretty quickly, even if you have 64GB+ of VRAM.

It depends what you use them for.

If you ask it for knowledge, like a comparison of vacuum cleaner models then yes, it's a hallucination fest. They just don't have the parameters for this level of detail. This is where ChatGPT is really king.

But if you give them the data they need with RAG, they're not bad. Acting on commands, looking stuff up in provided context, summarising all perform pretty well. Which seems to be also what Apple is targeting to do with them.

Re: Apple's On-Device and Server Foundation Models

#215
post #84

Is it me or Apple is really moving fast? I don't think it is easy for a company of this size to concisely put a vision of AI in these short and crazy AI times. BTW, not an Apple fan but an Apple user.

ChatGPT came out November 2022 and it took Apple 18 months to announce Siri will integrate with it.

Is that moving fast? Maybe, compared to what, Oracle?

Re: Apple's On-Device and Server Foundation Models

#216

Earlier quoted context omitted.

"AI for the rest of us."

Except Apple isn't really for the rest of us. Outside of America and a handful wealthy western countries it's for the top 5-20% earners only.

Japan and Taiwan are both more than 50% iOS.

Ref: https://worldpopulationreview.com/country-rankings/iphone-ma...

Re: Apple's On-Device and Server Foundation Models

#217
post #211
post #208

Earlier quoted context omitted.

Consider a feature from earlier in the keynote: the thing Notes (and Math Notes) does now where it fixes up your handwriting into a facsimile of your handwriting, with the resulting letters then acting semantically as text (snapping to a baseline grid; being reflowable; being interpretable as math equations) but still having the kind of long-distance context-dependent variations that can't be accomplished by just gen…

For everything other than handwriting I don't think the LoRAs are fine-tuned locally.

Well, here's another one: they promised that your local (non-iCloud) photos don't leave the device. Yet they will now — among many other things they mentioned doing with your photos — allow you to generate "Memoji" that look like the people in your photos. Which includes the non-iCloud photos.

I can't picture any way to use a RAG to do that.

I can picture a way to do that that doesn't involve any model fine-tuning, but it'd be pretty ridiculous, and the results would probably not be very good either. (Load a static image2text LoRA tuned to describe the subjects of photos; run that once over each photo as it's imported/taken, and save the resulting descriptions. Later, whenever a photo is classified as a particular subject, load up a static LLM fine-tune that summarizes down all the descriptions of photos classified as subject X so far, into a single description of the platonic ideal of subject X's appearance. Finally, when asked for a "memoji", load up a static "memoji" diffusion LoRA, and prompt it with the that subject-platonic-appearance description.)

But really, isn't it easier to just fine-tune a regular diffusion base-model — one that's been pre-trained on photos of people — by feeding it your photos and their corresponding metadata (incl. the names of subjects in each photo); and then load up that LoRA and the (static) memoji-style LoRA, and prompt the model with those same people's names plus the "memoji" DreamBooth-keyword?

(Okay, admittedly, you don't need to do this with a locally-trained LoRA. You could also do it by activating the static memoji-style LoRA, and then training to produce a textual-inversion embedding that locates the subject in the memoji LoRA's latent space. But the "hard part" of that is still the training, and it's just as costly!)

Re: Apple's On-Device and Server Foundation Models

#218

For people interested in AI research, there's nothing new here. IMO they should do a better job of referencing existing papers and techniques. The way they wrote about "adaptors" can make it seem like it's something novel, but it's actually just re-iterating vanilla LoRA. It was enough to convince one of the top-voted HackerNews comments that this was a "huge development". Benchmarks are nice though.

This gives me the vibe of calling high resolution screens as "retina" screens.

I don't see anything wrong with that at all. They've created a branding term that allows consumers to get an idea of the sort of pixel density they can expect without having to actually check, should they not want to bother.

Re: Apple's On-Device and Server Foundation Models

#219

Earlier quoted context omitted.

It's a huge development in terms of it being a consumer-ready, on-device LLM. And if Karpathy thinks so then I assume it's good enough for HN: https://x.com/karpathy/status/1800242310116262150

[flagged]

The temp around Elon here is lower than you think. I would say almost the exact opposite of your claim.

Re: Apple's On-Device and Server Foundation Models

#220
post #116

Earlier quoted context omitted.

The Llama 3 license says: "If, on the Meta Llama 3 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights unde…

They can just run the same tests and cite the results from other websites. That has nothing to do with Meta. No companies can force you to not talk about them.

The tests they ran were very different from what's usually run, mostly involving perception of usefulness to humans. I don't see what website they would've cited from?
Post reply on HN