Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Core AI Framework

developer.apple.com

Post reply on HN
Print view View original

21–30 of 114 posts

Re: Apple Core AI Framework

#21

Post by ActorNightly »

an0malous wrote: post #9

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

Very false.

I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited.

Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

Re: Apple Core AI Framework

#22

Post by davnicwil »

ActorNightly wrote: post #21
an0malous wrote: post #9

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

Very false. I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited. Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

well to be fair that's right now, I think the question is what about in 6 months, 12 months, 2 years?

Where do these improvement curves go? Does the gap close, do they intersect for practical purposes (factoring in cost etc)? Or is the local curve always just a translation of the hosted, lagging behind, or indeed does hosted just pull ahead?

Nobody knows, but it's a very open question I feel, and it certainly appears like the answer might quite reasonably be that yes they intersect on that kind of short-ish term time horizon.

Re: Apple Core AI Framework

#23

Post by XenophileJKO »

mindwok wrote: post #20
sealeck wrote: post #11

Earlier quoted context omitted.

Have we reached the limits of scaling? Sadly it appears that larger model still equals better model

I think GPT 4.5 showed that there is indeed a practical limit we're close too. That was supposedly a high-trillions of parameter model that was deprecated almost immediately because it was slow, insanely expensive, and had questionable benefits over the smaller models. Though apparently the new Mythos and whatever GPT Spud is (if it wasn't 5.5) are back up in the high trillions.

Actually having used it a bit, I'm quite excited to see a modern model of similar size.

I think what people didn't realize was, just because the GPT-4.5 model didn't get better on the benchmarks, didn't mean the model wasn't different than the earlier models. It was being compared to thinking models that were being developed at the same time.

The GPT 4.5 model still has some of the most "human" like abilities in communication even though it isn't particularly good a problem solving. It hadn't under gone the same type of reinforcement training.

I still use GPT 4.5 sometimes, in creative exercises it can be surprisingly effective. The model is still available.

Re: Apple Core AI Framework

#24

Post by LoganDark »

pzo wrote: post #5
bensyverson wrote: post #2

Wow, this seems to be a new way to convert PyTorch models to a format that runs across CPU, GPU & Apple's Neural Engine (ANE). [0] Does this completely replace the previous API, CoreML? [1] [0]: https://apple.github.io/coreai-optimization/ [1]: https://developer.apple.com/documentation/coreml/

seems they planning to replace it but overall now I'm really confused about this and mlx and coremltools. They should do better work explaining the benefits (and cons) of it and any feature parity between coreai, coreml and mlx.

My reading of it is:

- Core ML is for models designed only for Apple platforms

- MLX is for models that don't need to be fast

- Core AI is for models that run everywhere already and also need to be fast

Re: Apple Core AI Framework

#25

Post by wtallis »

criddell wrote: post #18

Is there something like this on Linux? For example, if I’m an application developer can I assume GNU Core AI (or whatever it is or would be called) will be there if the kernel is >= some particular version?

On non-Apple platforms, you generally have at least 2+(number of supported silicon vendors) different AI frameworks to worry about. I guess Apple's there now too, between Core ML, MLX, Core AI.

I haven't seen any sign that the framework fragmentation problem is going away anytime soon. NVIDIA wants everyone to do all training and inference with CUDA and to deny that NPUs have any usefulness. Everybody making an NPU has a different framework tailored to their architecture and the limitations they inherited from hardware designed before LLMs existed, and most of them have a another framework for targeting a GPU. And the OS vendor has one or two frameworks they would prefer you use rather than something hardware-specific.

Re: Apple Core AI Framework

#26

Post by ActorNightly »

davnicwil wrote: post #22
ActorNightly wrote: post #21

Earlier quoted context omitted.

Very false. I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited. Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

well to be fair that's right now, I think the question is what about in 6 months, 12 months, 2 years? Where do these improvement curves go? Does the gap close, do they intersect for practical purposes (factoring in cost etc)? Or is the local curve always just a translation of the hosted, lagging behind, or indeed does hosted just pull ahead? Nobody knows, but it's a very open question I feel, and it certainly appears…

>Where do these improvement curves go?

Nowhere.

Large models haven't seen that much improvement, just small unique tasks performance which is all special cased RLed to game metrics

For local models, its the same story. You can download Gemma 3 QAT from last year, and it will be just as good as Gemma:31b on the average. Qwen also boasts that its better, because again, they RLed it to game some metrics. Its better in coding then Gemma, but Gemma is better in more creative thinking (again, all RL)

Fundamentally, you need detail in the gradients for the models to pick up on the smaller details. If you don't have those, your output is gonna suck. No amount of clever architecture is going to fix this.

The only way to improve local models by training them to fetch context, and then their job becomes much simpler because all they need to do is reinterpret the fetched content and provide an answer. But fundamentally, if you are trying to keep things in house for advertising purposes like what all companies do with search, you want them to go to your service, which means running on your servers. And its not really that much extra per invocation (i.e excluding initial hardware costs) to instead just offer a large model as a service, which will be way better than any small models.

Re: Apple Core AI Framework

#27

Post by dvt »

AI future is clearly local, and my recent pitch has been "infinite tokens." Because that's what my M1 MBP can do; and that's what my RTX3090 can do. I don't need to pay hundreds of dollars a month and no one else does either.

Re: Apple Core AI Framework

#28

Post by dvt »

ActorNightly wrote: post #21
an0malous wrote: post #9

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

Very false. I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited. Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

> Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

On a price-per-wattage level, this is not true, people have done the math on /r/LocalLLaMA many times over[1]. Local models, while not as good as premier models (GPT 5.5, etc.), are like ~80%+ of the way there, and often converge to a similar solution after a few dead ends.

[1] https://www.reddit.com/r/LocalLLM/comments/1kshq4f/electrici...

Re: Apple Core AI Framework

#29

Post by fwip »

dvt wrote: post #28
ActorNightly wrote: post #21

Earlier quoted context omitted.

Very false. I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited. Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

> Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models. On a price-per-wattage level, this is not true, people have done the math on /r/LocalLLaMA many times over[1]. Local models, while not as good as premier models (GPT 5.5, etc.), are like ~80%+ of the way there, and often converge to a similar solution after a few dead ends. [1] https://www.reddit.com/r/LocalLLM/comments/1ksh…

Maybe not per watt, but unless you already happen to own a 3900 cited by that post, you'd have to buy that as well, which is currently selling for around $1400 used.

Re: Apple Core AI Framework

#30

Post by viccis »

an0malous wrote: post #9

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

I just want a tiny tiny model that runs on device that knows for autocomplete that, for example, I want to say "I'll be right back" instead of "I'll be right Brian". That's my #1 AI ask right now. Please, Apple.
Post reply on HN