Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MLX: An array framework for Apple Silicon

github.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: MLX: An array framework for Apple Silicon

#31

Post by beeboobaa »

sgu999 wrote: post #14
babl-yc wrote: post #9

I've found the process of porting custom ML models to iOS extremely difficult. AFAIK the only way to leverage Apple Neural Engine (and get the best performance) is to use CoreML. The only documented way to use CoreML is via coremltools, which takes a trace of a PyTorch model and attempts to translate it into a protobuf graph understood by CoreML. This process often fails and requires model changes, or worse "succeeds…

I don't understand why Apple isn't trying to integrate better with the standard tools for that field. I guess it makes sense to lock in app devs, but ML eng.? That said I've had good success with onnxruntime recently [0]. [0] https://onnxruntime.ai/docs/execution-providers/CoreML-Execu...

It's apple. If they can't lock you in and milk you it's not worth doing.

Re: MLX: An array framework for Apple Silicon

#32

Post by jbarrow »

I'm not certain how many other researchers would be willing to swap away from PyTorch (or tf or jax) to yet another framework. Amazon pushed MXNet for a while, but the only time I used it was while interning there. From what I understand, advantage here is tight integration with Apple hardware/shared memory?

That being said, going to spend some time getting familiar with it by reimplementing Llama-2 and trying to make it fast: https://github.com/jbarrow/mlxllama

Re: MLX: An array framework for Apple Silicon

#34

Post by brrrrrm »

thanatosmin wrote: post #33

It looks like this is still missing many matrix operations like QR, SVD, einsum, etc. Is there a clear route to using these on the GPU in Python on Apple Silicon? Last I checked the PyTorch backend was still missing at least QR...

factorization methods are somewhat uncommonly used in deep learning (the likely target of this framework) and have compute properties (such as approximate outputs, non-deterministic number of iterations) that make them unlike the BLAS++ standard APIs.

einsum seems like a reasonable thing to request, but it's hard to be performant across the entire surface exposed by the operation.

Re: MLX: An array framework for Apple Silicon

#35

Post by thanatosmin »

brrrrrm wrote: post #34
thanatosmin wrote: post #33

It looks like this is still missing many matrix operations like QR, SVD, einsum, etc. Is there a clear route to using these on the GPU in Python on Apple Silicon? Last I checked the PyTorch backend was still missing at least QR...

factorization methods are somewhat uncommonly used in deep learning (the likely target of this framework) and have compute properties (such as approximate outputs, non-deterministic number of iterations) that make them unlike the BLAS++ standard APIs. einsum seems like a reasonable thing to request, but it's hard to be performant across the entire surface exposed by the operation.

Exactly right that this targets a narrower surface to enable many deep learning models. I wonder how uncommon it is to hit some operation that is not included, though? It seems pretty common from a PyTorch MPS tracking issue:

https://github.com/pytorch/pytorch/issues/77764

NVIDIA's moat is not just in providing BLAS++ operations, but extending this to a wider range of cuSPARSE, cuSOLVE, cuTENSOR, etc. Without these, it feels like Apple is just trying to play catch up with whatever is popular and unsupported...

Re: MLX: An array framework for Apple Silicon

#37

Post by docfort »

domschl wrote: post #17
sgu999 wrote: post #14

Earlier quoted context omitted.

I don't understand why Apple isn't trying to integrate better with the standard tools for that field. I guess it makes sense to lock in app devs, but ML eng.? That said I've had good success with onnxruntime recently [0]. [0] https://onnxruntime.ai/docs/execution-providers/CoreML-Execu...

The project probably at least partially serves as documentation for other platforms to integrate Silicon acceleration. It basically demonstrates how to use macOS Accelerate and Metal MPS (metal performance shaders) using C++ for Machine Learning and training optimization. Thus other platforms can simply take this backend-code and integrate it. (Pytorch basically did that already with Apple's help).

It doesn’t use MPS, so at least that part is more interesting than the usual approach.

Re: MLX: An array framework for Apple Silicon

#38

Post by lamontcg »

domschl wrote: post #29
sgu999 wrote: post #15

Earlier quoted context omitted.

Note that there is a Metal backend for PyTorch [0]. Sadly it doesn't work well with codebases that didn't account for it from the start... [0] https://developer.apple.com/metal/pytorch/

This is (partly) outdated. MPS (metal performance shaders) are now (since torch 2.x) fully integrated in standard Pytorch releases, no external backends or special torch versions are needed. There are few limitations left when compared with other backends. Instead of using 'cuda' device, one simply uses 'MPS' as device. What remains is: the optimizations Pytorch provides (especially compile() with 2.1) focus on cuda…

Practically does using unified memory mean that the slow transfer of training/testing data to the GPU would be eliminated?

Re: MLX: An array framework for Apple Silicon

#39

Post by GeekyBear »

kiratp wrote: post #7

I wish there was more information as to how this differs from or improves on Jax. Flax+Jax+OpenXLA seems to finally be building some momentum so when a big player launches yet another competitor, the justification for it would be a good thing to see. What was “not good enough” with Jax? Why did it make sense to put this human time and energy there instead of doubling down on Flax/Jax/OpenXLA? How will this move the n…

> I wish there was more information as to how this differs from or improves on Jax.

The quick start guide has an overview.

> The Python API closely follows NumPy with a few exceptions. MLX also has a fully featured C++ API which closely follows the Python API.

The main differences between MLX and NumPy are:

        Composable function transformations: MLX has composable function transformations for automatic differentiation, automatic vectorization, and computation graph optimization.

        Lazy computation: Computations in MLX are lazy. Arrays are only materialized when needed.

        Multi-device: Operations can run on any of the supported devices (CPU, GPU, …)
The design of MLX is inspired by frameworks like PyTorch, Jax, and ArrayFire. A noteable difference from these frameworks and MLX is the unified memory model. Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device types without performing data copies. Currently supported device types are the CPU and GPU.

https://ml-explore.github.io/mlx/build/html/index.html

Re: MLX: An array framework for Apple Silicon

#40

Post by compinter »

apstats wrote: post #2

This is really cool. I wonder how long it will be till we have GPT-4 quality models that run locally (if we ever will). Would open up a lot of possibilities.

We aren't quite there yet, but the last year has been an incredibly exciting time for the open source LLM community. If your computer is decently powerful, you might be really surprised by what's already possible. LM Studio on Apple Silicon supports GGUF quants on GPU out the box.
Post reply on HN