[deleted]
My iPhone 16 Pro Max produces garbage output when running MLX LLMs
201–210 of 239 posts
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#202Methodology is one thing; I can't really agree that deploying an LLM to do sums is great. Almost as hilarious as asking "What's moon plus sun?" But phenomenon is another thing. Apple's numerical APIs are producing inconsistent results on a minority of devices. This is something worth Apple's attention.
(This is a total digression, so apologies) My mind instantly answered that with "bright", which is what you get when you combine the sun and moon radicals to make 明( https://en.wiktionary.org/wiki/%E6%98%8E ) Anyway, that question is not without reasonable answers. "Full Moon" might make sense too. No obvious deterministic answer, though, naturally.
In the game Clair Obscur sun plus moon equals twilight.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#203Earlier quoted context omitted.
So the underlying issue is that the iPhone 16 Pro SKU was misdetected as having Neural Accelerator (nax) support and this caused silently wrong results. Not a problem with the actual hardware.
From a debugging point of view, the author's conclusion was still completely reasonable given the evidence they had
No it wasn't. A hardware defect so disastrous that it affects floating point computation on the neural engine, yet so minor that it does not affect any of the software on the device utilizing that hardware is exceedingly improbable.
The conclusion, that it was not the fault of the developer was correct, but assuming anything other than a problem at some point in the software stack is unreasonable.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#204Earlier quoted context omitted.
Reinstalling the OS sucks. I need to pull all my bank cards out of my safe and re-add their CVV's to the wallet, and sometimes authenticate over the phone. And re-register my face. And log back in to all my apps. It can take an hour or so, except it's spread out over weeks as I open an app and realize I need to log in a dozen times.
> And log back in to all my apps Isn’t this built in when transferring devices? Are backups different?
Some information stored in Secure Enclave and cannot be pulled out from there by design.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#205Methodology is one thing; I can't really agree that deploying an LLM to do sums is great. Almost as hilarious as asking "What's moon plus sun?" But phenomenon is another thing. Apple's numerical APIs are producing inconsistent results on a minority of devices. This is something worth Apple's attention.
(This is a total digression, so apologies) My mind instantly answered that with "bright", which is what you get when you combine the sun and moon radicals to make 明( https://en.wiktionary.org/wiki/%E6%98%8E ) Anyway, that question is not without reasonable answers. "Full Moon" might make sense too. No obvious deterministic answer, though, naturally.
You could play Infinite Craft and find out what the game thinks it is:
https://neal.fun/infinite-craft/
Edit: Spoiler -
It's 'Eclipse'
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#206Earlier quoted context omitted.
From a debugging point of view, the author's conclusion was still completely reasonable given the evidence they had
No it wasn't. A hardware defect so disastrous that it affects floating point computation on the neural engine, yet so minor that it does not affect any of the software on the device utilizing that hardware is exceedingly improbable. The conclusion, that it was not the fault of the developer was correct, but assuming anything other than a problem at some point in the software stack is unreasonable.
> yet so minor that it does not affect any of the software on the device utilizing that hardware
You're being unfair here. The showpiece software that uses that hardware wouldn't install, and almost all software ignores it.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#207The author is assuming Metal is compiled to ANE in MLX. MLX is by-and-large GPU-based and not utilizing ANE, barring some community hacks.
ANE is probably the biggest scam "feature" Apple has ever sold.
>ANE is probably the biggest scam "feature" Apple has ever sold.
It is astonishing how often ANE is smeared on here, largely by people who seem to have literally zero idea what they're talking about. It's often pushed by either/or people who bizarrely need to wave a flag.
MLX doesn't use ANE for the single and only reason that Apple hid the ANE behind CoreML, exposing zero public APIs to utilize ANE directly, and MLX -- being basically an experimental grounds -- wanted to hand roll their implementation around the GPU / CPU. They literally, directly state this as the reason. People inventing technical reasons for why MLX doesn't use ANE are basically just manufacturing a fan fiction. This isn't to say that ANE would be suitable for a lot of MLX tasks, and it is a highly optimized, power-efficient inference hardware that doesn't work for a lot of purposes, but its exclusion is not due to technically unsuitability.
Further, the ANE on both my Mac and my iPhone is constantly attenuating and improving my experience. Little stuff like extracting contents from images. Ever browse in Safari and notice that you can highlight text in the image almost instantly after loading a page? Every image, context and features detected effortlessly. Zero fans cycling up. Power usage at a trickle. It just works. It's the same way that when I take a photo I can search "Maine Coon" and get pictures of my cats, ANE used for subject and feature extraction. Computational photography massively leverages the ANE.
At a trickle of power.
Scam? Yeah, I like my battery lasting for more than a couple of minutes.
Apple intended ANE to bring their own NN augmentations to the OS and thus the user experience, and even the availability in CoreML as a runtime engine is more limited than what Apple's own software can do. Apple basically limits the runtime usage to ensure that no third party apps inhibit or restrict Apple's own use of this hardware.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#208Earlier quoted context omitted.
I guess at the bit level, but not at the level of computation? Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory.
> Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory. Why? This is well specified by IEEE 754. Many runtimes (e.g. for Javascript) use NaN boxing. Treating floats as a semi-arbitrary selection of rational numbers plus a handful of special values is /more/ correct than treating them as real numbers, but treating them as actually specified does giv…
> Many runtimes (e.g. for Javascript) use NaN boxing.
But I've never seen them depend on those NaNs surviving the FPU. Hell, they could use the same trick on bit patterns that overlap with valid float values if they really wanted to.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#209Earlier quoted context omitted.
No it wasn't. A hardware defect so disastrous that it affects floating point computation on the neural engine, yet so minor that it does not affect any of the software on the device utilizing that hardware is exceedingly improbable. The conclusion, that it was not the fault of the developer was correct, but assuming anything other than a problem at some point in the software stack is unreasonable.
> yet so minor that it does not affect any of the software on the device utilizing that hardware You're being unfair here. The showpiece software that uses that hardware wouldn't install , and almost all software ignores it.
The hardware itself is utilized by many pieces of software on any Apple device. Face ID uses it, Siri uses it, the camera uses it, there are also other Apple on device LLM features, where you could easily test whether the basic capabilities are there.
I highly doubt that you could have a usable iPhone with a broken neural engine, at the very least it would be obvious to the user that there is something very wrong going on.
Re: My iPhone 16 Pro Max produces garbage output when running MLX LLMs
#210Earlier quoted context omitted.
From a debugging point of view, the author's conclusion was still completely reasonable given the evidence they had
No it wasn't. A hardware defect so disastrous that it affects floating point computation on the neural engine, yet so minor that it does not affect any of the software on the device utilizing that hardware is exceedingly improbable. The conclusion, that it was not the fault of the developer was correct, but assuming anything other than a problem at some point in the software stack is unreasonable.
> The conclusion, that it was not the fault of the developer was correct, but assuming anything other than a problem at some point in the software stack is unreasonable.
Aah, the old "you're holding it wrong" defense.