Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Core AI Framework

developer.apple.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Apple Core AI Framework

#81

Post by niek_pas »

mikestorrent wrote: post #19
sealeck wrote: post #11

Earlier quoted context omitted.

Have we reached the limits of scaling? Sadly it appears that larger model still equals better model

Well, let's not forget that text models are not the only models! Video models are much slower and need comparatively more resources, and all they can do even at that size is generate videos a few seconds long. Clearly a ton more work is going to go into those, and demand for them will probably increase as more creative tools get authored using them as a central part of the workflow. Low-res local rendering for previe…

This is definitely a good point. I imagine the max capacity for video models is significantly lower than for text models (there just aren't as many professionals in video as there are people who write text or code) but I could be wrong.

Re: Apple Core AI Framework

#82

Post by AdamN »

fedeb95 wrote: post #76
dvt wrote: post #27

AI future is clearly local, and my recent pitch has been "infinite tokens." Because that's what my M1 MBP can do; and that's what my RTX3090 can do. I don't need to pay hundreds of dollars a month and no one else does either.

the real money is in the coding surrounding models to make them efficient at specialized tasks. Casual users want general purpose models, and AI chat apps will stay for them. Most programs can benefit from a specialized AI that can be local, and #programs >> #users.

Also context - there's alot of context out there and it's faster to get it from servers.

It doesn't matter how good the model is if it doesn't have context from data sources.

Re: Apple Core AI Framework

#83

Post by ABS »

pzo wrote: post #5
bensyverson wrote: post #2

Wow, this seems to be a new way to convert PyTorch models to a format that runs across CPU, GPU & Apple's Neural Engine (ANE). [0] Does this completely replace the previous API, CoreML? [1] [0]: https://apple.github.io/coreai-optimization/ [1]: https://developer.apple.com/documentation/coreml/

seems they planning to replace it but overall now I'm really confused about this and mlx and coremltools. They should do better work explaining the benefits (and cons) of it and any feature parity between coreai, coreml and mlx.

looks to me like the docs don't give a feature-parity table, but they do draw the "role" lines once you read across them:

- Core ML narrows to classic, non-neural ML (its own docs now point you there for "decision trees or tabular feature engineering")

- Core AI takes neural nets and transformers (the new .aimodel format, the new profiler)

- MLX stays the separate bring-your-own-weights track (its WWDC sessions draw no line back to Core AI at all)

coreai-opt is the successor to coremltools on the optimization side.

Re: Apple Core AI Framework

#84

Post by ABS »

ankit219 wrote: post #48

they are also working on activations (w4a8, w4a16 from what i know). if they deliver (and a big if), it means that given their market reach, they can dictate the way sub 100b parameter models are trained and served to a large extent, given their major usecase would be on device (macos and not ios for most of them).

[flagged]

Re: Apple Core AI Framework

#85

Post by ABS »

robgough wrote: post #40
franze wrote: post #7

i am more excited about the ondevice foundation model update that is coming https://developer.apple.com/documentation/updates/foundation... (not much info yet) but i maintain https://github.com/Arthur-Ficial/apfel so i might be biased

Have you seen that they've added an `fm` tool? It was mentioned in the Platforms State of the Union. Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...

[dead]

Re: Apple Core AI Framework

#86

Post by ABS »

something I haven't seen highlighted anywhere yet, while I find it very interesting, is the distributed inference across Macs (JACCL over Thunderbolt 5), an OpenAI-compatible mlx_lm.server, agentic-on-Mac.

Apple keeps MLX (bring-your-own-weights) separate from Foundation Models / Core AI.

Re: Apple Core AI Framework

#87

Post by robgough »

harrouet wrote: post #68
robgough wrote: post #40

Earlier quoted context omitted.

Have you seen that they've added an `fm` tool? It was mentioned in the Platforms State of the Union. Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...

Wait, is PPC open bar ?

They've said there are limits, and increased limits for those on iCloud+ ... so it seems that Apple is in the selling LLM access game now. I don't think there are any details yet on the nature of those limits, and whether they can be increased as required etc.

Re: Apple Core AI Framework

#88

Post by weirdcat »

dofm wrote: post #72
horsawlarway wrote: post #41

Earlier quoted context omitted.

I want to echo this. I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware. I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf. Combined with Pi as a harness, and I'm surprised to find that it feels about as capable…

I have come at this at a slightly different angle. I am a fully-burned-out freelancer (in the last couple of years so severely and totally that I thought I had early onset dementia, and I am still not sure I don't). I don't really have an off-ramp to anything else yet, but the sea-change in the industry has been contributing to my feeling that I should knock it on the head. I must get past broad understanding of AI t…

How are you running that GGUF, and how many tokens/sec are you getting without MTP? My M1 Max gives me 65 t/s for non-MTP unsloth/gemma-4-26B-A4B-it-qat-GGUF (UD-Q4_K_XL), but with MTP that actually goes down to 56 t/s (at 63% accepted drafts).

Re: Apple Core AI Framework

#89

Post by iwontberude »

horsawlarway wrote: post #41
hadlock wrote: post #15

Earlier quoted context omitted.

Qwen's ~30B-class models are genuinely good enough for use if you can find a machine with enough memory bandwidth to run them at 30-90 tokens/second. It's been extremely telling that Qwen stopped releasing 120b class models. At some point in the next 10 years (maybe 3?) someone is going to release an Opus 4.5 class 256B model you can run locally. Right now our engineers use about $800/mo worth of opus tokens; at that…

I want to echo this. I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware. I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf. Combined with Pi as a harness, and I'm surprised to find that it feels about as capable…

I was freaked out being stuck with OpenAI and Anthropic. I setup qwen3.6:35b-mlx on my Mac Studio M1 Ultra and was blown away really. I am no longer afraid that Anthropic or OpenAI will be able to control the market.

Re: Apple Core AI Framework

#90

Post by iwontberude »

maxdo wrote: post #45
horsawlarway wrote: post #41

Earlier quoted context omitted.

I want to echo this. I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware. I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf. Combined with Pi as a harness, and I'm surprised to find that it feels about as capable…

Majority of my agentic setup is pi / Claude code where every single Chinese models are not as good except commercial 1T models . Local is a pipe dream . If you can run it cheap occasionally why commercial companies can’t run it cheaper 24/7 and lower the costs ? The answer is simple. Use cases are more demanding and hence you need more from model not less . Sure if you task is to do a narrow labeling task on 1m recor…

Keep working on your agentfu because there is a sweet spot with subagents and parallelizable plans. It’s not about better, it’s about efficiency and picking the right model for the job. You can achieve the same results as frontier models with the right type of planning and context management on local Chinese models.
Post reply on HN