21–30 of 30 posts
It does not seem to cover the Neural Accelerators, Apple's equivalent of the Tensor Cores. They only got released on M5 platform. This is probably the most important part to cover.
Is there a non-slop version of this information available? I am reading up on GPU / ML micro architecture and am looking for some good sources.
https://news.ycombinator.com/item?id=47208573 Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering (maderix.substack.com) 376 points | 3 months ago | 122 comments
Really wish this author would blog more, this piece is incredible and includes the code.
Also ModernBERT is amazing if you haven’t used it before, worth spending time with - have used it myself for classification tasks and it’s very impressive.
Is there a non-slop version of this information available? I am reading up on GPU / ML micro architecture and am looking for some good sources.
I skimmed through it, what makes you think it is slop?
I did however struggle to run a diffusion model on the ANE - but found that mlx-swift and iPhone GPU sufficed: https://www.duration.ai/blog/generating-images-with-a-2020-i...
This Neural Engine seems useless for LLMs. Trapped in the wrong architecture
Apple is releasing CoreAI which is supposed to be optimized for LLMs and the transformer architecture.
If anyone is interested in doing something seriously useful with these neural cores, there is this incredible write up on getting ModernBERT running on them: https://stephenpanaro.com/blog/modernbert-on-apple-neural-en... Really wish this author would blog more, this piece is incredible and includes the code. Also ModernBERT is amazing if you haven’t used it before, worth spending time with - have used it myself for…
Earlier quoted context omitted.
Apple is releasing CoreAI which is supposed to be optimized for LLMs and the transformer architecture.
I've done some basic testing of the CoreAI framework (using Apple's official 'llm-runner' and officially supported .coreai converted models) and seen no noticable performance increase between standard MLX or GGUF with llama.cpp. I'd love to see some thorough benchmarks from someone though.
This scans very much as AI-written.
This is obvious Claude slop writing, the author would be advised to use vale [1] with samples of their own writing as a guide. > Performance begins with the roofline. On the M1 the engine holds about 12 fp16 TFLOP/s of compute against a DRAM-bandwidth ceiling. The roofline has a ridge point near 141 FLOP per byte, a 2 MB working-set threshold, a 0.23 ms floor under any single dispatch, and efficiency near 0.37 picojo…