Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's On-Device and Server Foundation Models

machinelearning.apple.com

Post reply on HN
Print view View original

361–370 of 562 posts

Re: Apple's On-Device and Server Foundation Models

#361

Post by p_l »

glial wrote: post #228
wraptile wrote: post #224

Earlier quoted context omitted.

RAM is literally the cheapest primary component in a laptop at going rate of 1-4usd/GB. I'd say that shipping 8GB base model in 2024 is clearly manipulation by Apple, i.e. planned obsolescence or a way to moat Apple software. Anyone who doesn't see this is just being delusional. Same way Apple and Samsung ship 128GB of storage when the production price between 128gb and 1tb is like 10$ (on a 1000$ device). Samsung ev…

> RAM is literally the cheapest primary component Is that still true for Apple's integrated memory? It might be - I just don't know.

The memory used by Apple isn't anything magical or special - it's bog standard LPDDR5, essentially same as phone - and in a laptop it's way less limited by thermal and power constraints to add more (which is how you have the rather large possible set of options).

While going for the top tier of memory sizes Apple offers does cost considerable amounts, making 16, or even 32GB standard is peanuts.

Re: Apple's On-Device and Server Foundation Models

#363

Post by flakiness »

gokuldas011011 wrote: post #247
ra7 wrote: post #34

> Our foundation models are trained on Apple's AXLearn framework, an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. Interesting that they’re using TPUs for training, in addition to GPUs. Is it both a technical decision (J…

"Use the best tool available"

They hired people nearby. Conveniently there is a small town called Mountain View.

Re: Apple's On-Device and Server Foundation Models

#364

Post by jacooper »

boxed wrote: post #359
orbital-decay wrote: post #337

> 2. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models. How do they represent users around the globe authentically while being located in Cupertino, CA? (more of a rhetorical question really)

I wish I could have one keyboard on my iPhone and could type both Swedish and English with it. These are the basics they can't get right, and I don't see why. They clearly have bilingual people working over there, why is this so bad?

Because then you will be typing wrong/s

Re: Apple's On-Device and Server Foundation Models

#365

Post by sanxiyn »

bt1a wrote: post #316
wpm wrote: post #188

Earlier quoted context omitted.

AFAIK there is no general purpose, "do this on the ANE" API. You have to be using specific higher level APIs like CoreML or VisionKit in order for it to end up on the ANE.

This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

It works quite well as long as you don't care about battery.

Re: Apple's On-Device and Server Foundation Models

#366

Post by woadwarrior01 »

jjtheblunt wrote: post #190
woadwarrior01 wrote: post #144

Earlier quoted context omitted.

None of the current iOS and macOS LLM Apps use the Neural Engine. They use the CPU and the GPU. nb: I'm the author of a fairly popular app in that category.

How would you know none of the apple apps use the neural engine? Is the key in the statement “LLM”?

Yes, I specifically meant autoregressive LLMs. BERT style encoder only models, ViTs and CNNs ran perfectly fine. Yesterday's coremltools update[1] changes that.

[1]: https://github.com/apple/coremltools/pull/2232

Re: Apple's On-Device and Server Foundation Models

#367

Post by woadwarrior01 »

epipolar wrote: post #13

It would be interesting to see how these models impact battery life. I’ve tried a few local LLMs on my iPhone 15 Pro via the PrivateLLM app, and the battery charge plummets just after a few minutes of usage.

I'm the author of Private LLM. Looks like it's just become possible[1] to run quantized LLM inference using the ANE with iOS 18. I think there are some major efficiency gains on the table now.

[1]: https://github.com/apple/coremltools/pull/2232

Re: Apple's On-Device and Server Foundation Models

#368

Post by woadwarrior01 »

TheRoque wrote: post #33

Why isn't there a comparison with the Llama3 8b in the "benchmarks" ?

Because their model won't look good in comparison. Also see this part of the footnote: "The open-source and Apple models are evaluated in bfloat16 precision." The end user's on-device experience will be with a quantized model and not the bfloat16 model.

Re: Apple's On-Device and Server Foundation Models

#369

Post by pests »

snemvalts wrote: post #205
keyle wrote: post #154

Earlier quoted context omitted.

It probably will change. Note that, so far, a 16GB apple device has much better usability than the equivalent on windows. This may sound biased, but the memory compression and foreground/background actions by macOS tight integration with the hardware is really good. I've never felt like I couldn't do things on smaller hardware, except (larges) LLMs. Also when I compare with my co-workers the memory pressure is a lot…

The swapping is indeed faster as the SSD is on the SoC and so fast to access. To the point that an 4 year old 8gb M1 Air is enough for simpler development work, at least for me.

I would think any 4 year old 8gh laptop would be enough for simpler development work.

Re: Apple's On-Device and Server Foundation Models

#370

Post by scarface_74 »

binkethy wrote: post #320
overstay8930 wrote: post #174

Earlier quoted context omitted.

People who understand cybersecurity hygiene use iPhones for obvious reasons

It would seem that integrating a backdoor funnel to OpenAI is a bit of a security issue to those who care about such things. Yay, we can all train corporate models for free involuntarily. I guess it's time to check out Lineage OS and Postmarket OS. It was always a matter of time.

No one is going to train an AI on random user generated data. The data is going to be horrible and it’s going to be full of PII that’s too risky to expose.
Post reply on HN