Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Core AI Framework

developer.apple.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Apple Core AI Framework

#71

Post by sgt101 »

hparadiz wrote: post #57
tyre wrote: post #55

Earlier quoted context omitted.

Those are not GPUs available on iPhones. Will we get there eventually? Maybe! Maybe we end up with GPU clusters built on the edge (e.g. cell towers) for offloading, maybe it’s never economical, maybe a different model architecture makes it simpler, who knows. But it doesn’t seem anywhere imminent with our current world state.

My computer is 15,000 times faster and costs in inflation adjusted dollars half that of my computer in 1995. There's zero reason to think that won't happen over the next 30 years again. For whatever reason every generations thinks they are the peak. Naw man. You're just a blip at the bottom of the logarithmic chart.

For me there are a bunch of questions:

- was the pause in model scaling a result of the benefits of RL & SFT being easier to access and quicker than scaling, or was it genuinely the result of scaling being low ROI now?

- are power densities necessary to provide high quality on device inference possible? Can the best, technically feasible, architectures accomodate T scale models and run them off batteries that fit in your hand?

- will thing slow down enough to allow edge depoloyments to realise value vs. centralised deployments.

- do edge use cases drive enough revenue to get this to happen?

- can local inference make up for model scale? Does that make sense in a latency/power race with the central infrastructure? Is there a sweet spot here?

I am not sure about any of the answers...

Re: Apple Core AI Framework

#72

Post by dofm »

horsawlarway wrote: post #41
hadlock wrote: post #15

Earlier quoted context omitted.

Qwen's ~30B-class models are genuinely good enough for use if you can find a machine with enough memory bandwidth to run them at 30-90 tokens/second. It's been extremely telling that Qwen stopped releasing 120b class models. At some point in the next 10 years (maybe 3?) someone is going to release an Opus 4.5 class 256B model you can run locally. Right now our engineers use about $800/mo worth of opus tokens; at that…

I want to echo this. I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware. I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf. Combined with Pi as a harness, and I'm surprised to find that it feels about as capable…

I have come at this at a slightly different angle.

I am a fully-burned-out freelancer (in the last couple of years so severely and totally that I thought I had early onset dementia, and I am still not sure I don't). I don't really have an off-ramp to anything else yet, but the sea-change in the industry has been contributing to my feeling that I should knock it on the head.

I must get past broad understanding of AI to deep understanding, but I have to find a way to do this which sits well with freelancer ethics (sustainability, stability, control of destiny).

So I decided I would start out with that operating principle that ultimately this stuff is just going to be local: models will eventually hit some level of practicality for most tasks and technological progress guarantees that they will eventually run on desktops.

I decided to learn how to run models locally properly, see how far I get with opencode (and Pi and Zed experiments), and grow outwards from there to metered models (opencode go, openrouter etc.)

Knowledge first; what can I do that meaningfully changes my outcomes and confidence with no cost and no exposure to sudden change?

I have a secondhand M1 Max (excellent GPU bandwidth), and I am really shocked to find that arguably that level of practicality is already here.

Qwen 3.6 35B can really do a lot. And — not sure if you have tested it — but in some ways I think the Gemma 4 26B is better. Particularly for more commonplace dev tech — it is very knowledgeable about the sort of low-end web dev stack that is most common (Wordpress, PHP, MySQL).

I have been getting 75 tokens/sec with (GGUF) Gemma-4 26B QAT and MTP. (Can't get anywhere close with MLX, for some reason.)

A similar sort of speed with an MLX Qwen 3.6 35B. I have a sneaking suspicion that maybe llama.cpp is now faster than MLX on this older kit so I might try seeing what llama.cpp can do there, too.

Not blazing fast, but fast enough that there are plenty of experiments and small jobs I can do before I even get to using Big Pickle!

Re: Apple Core AI Framework

#73

Post by franze »

robgough wrote: post #40
franze wrote: post #7

i am more excited about the ondevice foundation model update that is coming https://developer.apple.com/documentation/updates/foundation... (not much info yet) but i maintain https://github.com/Arthur-Ficial/apfel so i might be biased

Have you seen that they've added an `fm` tool? It was mentioned in the Platforms State of the Union. Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...

did miss it until now, cool to see it on device and first party. as soon as it lands I will see the impact on apfel.

but i definitely feel flattered, either my little project inspired them or that I reached the same conclusion at a similar time as a team at apple that "hey, this is totally missing"

Re: Apple Core AI Framework

#74

Post by ClikeX »

fwip wrote: post #29
dvt wrote: post #28

Earlier quoted context omitted.

> Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models. On a price-per-wattage level, this is not true, people have done the math on /r/LocalLLaMA many times over[1]. Local models, while not as good as premier models (GPT 5.5, etc.), are like ~80%+ of the way there, and often converge to a similar solution after a few dead ends. [1] https://www.reddit.com/r/LocalLLM/comments/1ksh…

Maybe not per watt, but unless you already happen to own a 3900 cited by that post, you'd have to buy that as well, which is currently selling for around $1400 used.

To be fair, I can also use that 3900 for other things locally. Not just AI.

Re: Apple Core AI Framework

#75

Post by franze »

dofm wrote: post #70
robgough wrote: post #40

Earlier quoted context omitted.

Have you seen that they've added an `fm` tool? It was mentioned in the Platforms State of the Union. Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...

Is ‘fm serve’ OpenAPI compatible?

fm serve - "Start a Chat Completions API server"

chat completion is openai's api surface name.

but only when it is actually available we will see if it's a clean drop-in vs. just "chat-completions-ish".

one of my learnings from apfel is that is is very easy to get a kinda openAI api compatible server, and a lot of work to get it really totally compatible. sometimes i wonder if even the openai implementation of openai's api is openai api compatible to the core....

Re: Apple Core AI Framework

#76

Post by fedeb95 »

dvt wrote: post #27

AI future is clearly local, and my recent pitch has been "infinite tokens." Because that's what my M1 MBP can do; and that's what my RTX3090 can do. I don't need to pay hundreds of dollars a month and no one else does either.

the real money is in the coding surrounding models to make them efficient at specialized tasks. Casual users want general purpose models, and AI chat apps will stay for them. Most programs can benefit from a specialized AI that can be local, and #programs >> #users.

Re: Apple Core AI Framework

#77

Post by dofm »

franze wrote: post #75
dofm wrote: post #70

Earlier quoted context omitted.

Is ‘fm serve’ OpenAPI compatible?

fm serve - "Start a Chat Completions API server" chat completion is openai's api surface name. but only when it is actually available we will see if it's a clean drop-in vs. just "chat-completions-ish". one of my learnings from apfel is that is is very easy to get a kinda openAI api compatible server, and a lot of work to get it really totally compatible. sometimes i wonder if even the openai implementation of openai…

> chat completion is openai's api surface name

Ahh! I did not know that

> sometimes i wonder if even the openai implementation of openai's api is openai api compatible to the core….

It's a similar situation with "Arca-Swiss compatible" tripod plates in photography. There is really no such thing — Arca-Swiss didn't make a standard, so they didn't have to stick to it themselves, and while most things using this "standard" fix to most things, some things just won't fit, or won't stay put. Everyone implements it, and if they don't, people complain "why didn't you just put an Arca standard foot on it?" and then you have to sit them down and tell them.

Re: Apple Core AI Framework

#78

Post by romanovcode »

maxdo wrote: post #43
an0malous wrote: post #9

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

Why on earth I should switch from a top tier model to much worse local model ? Why do I need to suffer my battery ?

Right now there is no reason since tokens are subsidized heavily. However when OpenAI/Anthropic will drop the $200/month pricing since most likely it eventually will become unsustainable you'd rather get MacBook Pro M6 Ultra with 128GB ram and go local then pay thousands every month for tokens.

Re: Apple Core AI Framework

#79

Post by ActorNightly »

dvt wrote: post #32
fwip wrote: post #29

Earlier quoted context omitted.

Maybe not per watt, but unless you already happen to own a 3900 cited by that post, you'd have to buy that as well, which is currently selling for around $1400 used.

I do have a 3090 Ti on my gaming PC, but even my old M1 MBP (with a mere 32gb of RAM) is quite competent and can run a quantized `Gemma4-26B-A4B` in the background while I do other stuff.

The MBP running Gemma4 is absolutely is useless for any real work.

Re: Apple Core AI Framework

#80

Post by sgt »

wahnfrieden wrote: post #6
bensyverson wrote: post #2

Wow, this seems to be a new way to convert PyTorch models to a format that runs across CPU, GPU & Apple's Neural Engine (ANE). [0] Does this completely replace the previous API, CoreML? [1] [0]: https://apple.github.io/coreai-optimization/ [1]: https://developer.apple.com/documentation/coreml/

Requires OS 27+, so CoreML is still useful for backwards compatibility.

macOS users aren't that good at upgrading regularly, but iOS users are at least obsessive about upgrading to the latest OS. I guess the system almost forces us.
Post reply on HN