Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

github.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#31

Post by cl0ckt0wer »

gervwyk wrote: post #28

Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there.. Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach. Am i missing something fundamental?

llm intelligence seems to be proportional to the ram used. All techniques like this will be used by everyone.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#32

Post by [deleted] »

simonw wrote: post #21

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Downloa…

[deleted]

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#33

Post by simonw »

simonw wrote: post #21

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Downloa…

Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts):

  git clone --recursive https://github.com/SharpAI/SwiftLM.git
  cd SwiftLM

  swift build -c release

  # Trick to copy in that missing mlx.metallib file
  uv run --with mlx-metal python -c "
  import importlib.metadata, pathlib, shutil
  d = importlib.metadata.distribution('mlx-metal')
  metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib'
  shutil.copy(metallib, '.build/release/')
  print(f'Copied {metallib} -> .build/release/mlx.metallib')

  # Now start the server (downloads 69GB Qwen model)
  .build/release/SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413
But the server crashed when I tried to run a prompt through it:

  freed pointer was not the last allocation

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#34

Post by aegis_camera »

simonw wrote: post #33
simonw wrote: post #21

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Downloa…

Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts): git clone --recursive https://github.com/SharpAI/SwiftLM.git cd SwiftLM swift build -c release # Trick to copy in that missing mlx.metallib file uv run --with mlx-metal python -c " import importlib.metadata, pathlib, shutil d = importlib.metadata.distribution('mlx-metal') metallib = pathlib.Path(d._path).parent / 'mlx/l…

the Python mlx-metal trick is actually what's crashing it. The mlx.metallib from pip is a different version of MLX than what your Swift binary was built against. It gets past the startup error but then corrupts the GPU memory allocator at inference time → freed pointer was not the last allocation.

Use the version-matched metallib that's already in the repo:

cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/ .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 This is the exact metallib that was compiled alongside the Swift code — no version mismatch. Future pre-built releases will bundle it automatically.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#36

Post by daft_pink »

gervwyk wrote: post #28

Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there.. Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach. Am i missing something fundamental?

I’ve always believed local is the future. If you consider how your iPhone has a processor that is more powerful than something very large not too long ago.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#38

Post by aegis_camera »

daft_pink wrote: post #36
gervwyk wrote: post #28

Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there.. Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach. Am i missing something fundamental?

I’ve always believed local is the future. If you consider how your iPhone has a processor that is more powerful than something very large not too long ago.

I've ran this on an IPHONE 13 pro (6GB) memory, QWEN 3 1.7B runs good. So local will get more intelligent for the task you want it done soon or already.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#39

Post by aegis_camera »

altruios wrote: post #5
aegis_camera wrote: post #2

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To…

what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md

https://www.sharpai.org/benchmark/ The MLX part is what we've done with SwiftLM, the local result is still being verified more details are on-going.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#40

Post by aegis_camera »

anemll wrote: post #22
aegis_camera wrote: post #2

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To…

Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360

Thanks, pure Swift was the design idea and since I found nothing could be used for my project https://www.sharpai.org then I created Swift version. Python is too heavy to be delivered with application, user mentioned they want to use MLX, that's why I've been working on it for 1-2 weeks for bug fixing and testing , then suddenly TurboQuant proposed, I had a quick integration. My 64GB M5 Pro is already good for my local security task, now it's able to use M1/M2 Mini w/ 8GB memory.
Post reply on HN