Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My iPhone 16 Pro Max produces garbage output when running MLX LLMs

journal.rafaelcosta.me

Post reply on HN
Print view View original

171–180 of 239 posts

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#171

Post by xmcqdpt2 »

ekelsen wrote: post #109
addaon wrote: post #100

Earlier quoted context omitted.

> Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory. Why? This is well specified by IEEE 754. Many runtimes (e.g. for Javascript) use NaN boxing. Treating floats as a semi-arbitrary selection of rational numbers plus a handful of special values is /more/ correct than treating them as real numbers, but treating them as actually specified does giv…

Can you show me where in the ieee spec this is guaranteed? My understanding is the exact opposite - that it allows implementations to return any NaN value at all. It need not be any that were inputs. It may be that JavaScript relies on it and that has become more binding than the actual spec, but I don't think the spec actually guarantees this. Edit: actually it turns out nan-boxing does not involve arithmetic, which…

See 6.2.3 in the 2019 standard.

> 6.2.3 NaN propagation

> An operation that propagates a NaN operand to its result and has a single NaN as an input should produce a NaN with the payload of the input NaN if representable in the destination format.

> If two or more inputs are NaN, then the payload of the resulting NaN should be identical to the payload of one of the input NaNs if representable in the destination format. This standard does not specify which of the input NaNs will provide the payload.

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#172

Post by llm_nerd »

zozbot234 wrote: post #168
llm_nerd wrote: post #166

Earlier quoted context omitted.

Apple's documentation is utter garbage, but this code almost seems like a separate issue (and notably the MLX library uses loads of undocumented properties in metal which isn't cool). It looks like the change used to allow the NAX kernel to be used on the iPhone 17 or upcoming 18 if you're on 26.2 or later, to instead only allow it on the iPhone 17 Pro or upcoming 18. I'm fairly sure the GPU arch on the A19 is 17. Th…

> The neural accelerator exists in iPhones going back many years. What has existed before is the Apple Neural Engine (ANE) which is very different from the newer Neural Accelerator support within the GPU blocks. In fact MLX does not even support ANE yet since at least in previous versions it was hardware-limited to computing FP16 and INT8 MADDs, and not even that fast.

Sure, I directly and explicitly talked about Apple's version of tensor cores in the GPU. But the ANE is by every definition a neural accelerator. Yes, I'm aware of Apple's weird branding for their tensor cores.

"In fact MLX does not even support ANE yet"

I didn't say otherwise. The ANE is a fantastic unit for small, power-efficient models, like extracting text from images, doing depth modelling, etc. It's not made for LLMs, or the other sorts of experimental stuff MLX is intended for. Though note that MLX's author's reason for not supporting the ANE is that it has a "closed-source" API (https://github.com/ml-explore/mlx/issues/18#issuecomment-184...), making it unsuitable for an open-source project, and given that MLX didn't want to just lean on CoreML. But anyways, the ANE is fantastically fast at what it does, while sipping juice.

In any case, the code change shown should have zero impact on the running of MLX on an iPhone 16 Pro. MLX tries to really leverage platform optimizations so maybe another bifucation is making the wrong choice.

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#174

Post by throwaway132448 »

RulerOf wrote: post #150
post-it wrote: post #110

Earlier quoted context omitted.

Reinstalling the OS sucks. I need to pull all my bank cards out of my safe and re-add their CVV's to the wallet, and sometimes authenticate over the phone. And re-register my face. And log back in to all my apps. It can take an hour or so, except it's spread out over weeks as I open an app and realize I need to log in a dozen times.

There was a magical period. I suspect it ended with the introduction of the Secure Enclave. But maybe it was a little later. An encrypted iTunes backup of a device was a perfect image. Take the backup, pull the SIM card, restore the backup to a new phone with the sim card installed, and it was like nothing had happened. No reauthentication. No missing notifications. No lost data. Ever. It was nice.

Security theatre killed this. Everyone must be assumed to be a moron incapable of living with the consequences of their own choices at all times.

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#175

Post by dahcryn »

embedding-shape wrote: post #157
zcbenz wrote: post #156

It is a bug in MLX that has been fixed a few days ago: https://github.com/ml-explore/mlx/pull/3083

Blog post dated 28 Jan 2026, the bug fix posted 29 Jan 2026, so I guess this story had a happy ending :) Still, sad state of affairs that it seems like Apple is still fixing bugs based on what blog posts gets the most attention on the internet, but I guess once they started that approach, it's hard to stop and go back to figuring out priorities on their own.

I think you overestimate the power of a blogpost and the speed of bugfixing at Apple for something like this.

I almost guarantee there is no way they can read this blogpost, escalate it internally, get the appropriate approval to the work item, actually work on the fix, get it through QA and get it live in production in 3 days. That would only happen on really critical issues, and this is definitely not critical enough for that.

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#176

Post by zozbot234 »

llm_nerd wrote: post #172
zozbot234 wrote: post #168

Earlier quoted context omitted.

> The neural accelerator exists in iPhones going back many years. What has existed before is the Apple Neural Engine (ANE) which is very different from the newer Neural Accelerator support within the GPU blocks. In fact MLX does not even support ANE yet since at least in previous versions it was hardware-limited to computing FP16 and INT8 MADDs, and not even that fast.

Sure, I directly and explicitly talked about Apple's version of tensor cores in the GPU. But the ANE is by every definition a neural accelerator. Yes, I'm aware of Apple's weird branding for their tensor cores. "In fact MLX does not even support ANE yet" I didn't say otherwise. The ANE is a fantastic unit for small, power-efficient models, like extracting text from images, doing depth modelling, etc. It's not made fo…

The change's effects are dependent on what each SKU reports as its Metal architecture, both as identifying string (the equivalent to running 'metal-arch' in the Mac CLI) and as generation 'gen' number. Most likely you're misinterpreting the change as not affecting the iPhone 16 Pro, where in fact it does.

The MLX folks have various rationales for not supporting the ANE (at least as of yet), but one of them is that any real support requires implementing explicit splits in the graph of computations, where ANE-suitable portions are to be dispatched to the ANE and everything else goes back to the GPUs. That's not necessarily trivial.

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#178

Post by mrtesthah »

embedding-shape wrote: post #157
zcbenz wrote: post #156

It is a bug in MLX that has been fixed a few days ago: https://github.com/ml-explore/mlx/pull/3083

Blog post dated 28 Jan 2026, the bug fix posted 29 Jan 2026, so I guess this story had a happy ending :) Still, sad state of affairs that it seems like Apple is still fixing bugs based on what blog posts gets the most attention on the internet, but I guess once they started that approach, it's hard to stop and go back to figuring out priorities on their own.

How do you know that it wasn’t merely that the blog post elicited multiple people to file the same duplicate bug in Apple’s radar system, which is how they ostensibly prioritize fixes?

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#179

Post by TimByte »

zozbot234 wrote: post #160
zcbenz wrote: post #156

It is a bug in MLX that has been fixed a few days ago: https://github.com/ml-explore/mlx/pull/3083

So the underlying issue is that the iPhone 16 Pro SKU was misdetected as having Neural Accelerator (nax) support and this caused silently wrong results. Not a problem with the actual hardware.

From a debugging point of view, the author's conclusion was still completely reasonable given the evidence they had

Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs

#180

Post by TimByte »

csmantle wrote: post #46

Methodology is one thing; I can't really agree that deploying an LLM to do sums is great. Almost as hilarious as asking "What's moon plus sun?" But phenomenon is another thing. Apple's numerical APIs are producing inconsistent results on a minority of devices. This is something worth Apple's attention.

The scary part isn't "LLMs doing sums." It's that the same deterministic model, same weights, same prompt, same OS, produces different floating-point tensors on different devices
Post reply on HN