Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

github.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#21

Post by simonw »

I couldn't get the downloadable binary to work, or the binary I compiled myself:

  ./SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413
Error:

  [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit
  [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5
  [SwiftLM]  Memory strategy: FULL GPU (69.6GB model, 133.4GB available)
  [SwiftLM] Download: [===================>] 100% ⠋ (66395.4 MB / 66395.4 MB) | Speed: 0.0 MB/s      
  MLX error: Failed to load the default metallib. library not found library not found library not found library not found  at /Users/runner/work/SwiftLM/SwiftLM/LocalPackages/mlx-swift/Source/Cmlx/mlx-c/mlx/c/stream.cpp:115

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#22

Post by anemll »

aegis_camera wrote: post #2

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To…

Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#23

Post by sumeno »

yieldcrv wrote: post #11
brokencode wrote: post #9

Earlier quoted context omitted.

That’s a starting spot, but how about some testing and benchmarks? Where’s the value added if the person just tells Claude to do it and then submits a PR? The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

if it works it works we live in a wholly unoptimized world because the available resources have been so high, while the benefits of optimizing have been so low. that has flipped now and there are tons of low hanging fruit to optimize. I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

> if it works it works

If it works in one case that doesn't mean it works consistently or well in the general case

I've made lots of things with Claude Code that just work... until I do things in a slightly different order and the whole thing explodes

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#24

Post by [deleted] »

simonw wrote: post #21

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Downloa…

[deleted]

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#25

Post by pqtyw »

_zoltan_ wrote: post #7
Aurornis wrote: post #3

Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…

"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.

If there is nothing valuable it contributes, though? i.e. its not a novel paper then only value is the whatever you personally learn from it.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#26

Post by sumeno »

mjr00 wrote: post #8
_zoltan_ wrote: post #7

Earlier quoted context omitted.

"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.

> "vibe coded" is NOT the bad thing you think it is. It's not inherently bad in the same way that a first draft of a novel is not inherently bad. But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.

At least in the novel example the author had the decency to write what they're asking you to read.

These are more like sending someone who didn't ask you a question a LMGTFY link they didn't ask for and expecting them to read all the results. Just a complete lack of awareness and respect for the maintainers

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#27

Post by pqtyw »

yieldcrv wrote: post #11
brokencode wrote: post #9

Earlier quoted context omitted.

That’s a starting spot, but how about some testing and benchmarks? Where’s the value added if the person just tells Claude to do it and then submits a PR? The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

if it works it works we live in a wholly unoptimized world because the available resources have been so high, while the benefits of optimizing have been so low. that has flipped now and there are tons of low hanging fruit to optimize. I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

It might work, but what's the point is sharing it if anyone can do the same in those 30 minutes with minimal effort?

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#28

Post by gervwyk »

Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there..

Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach.

Am i missing something fundamental?

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#29

Post by Aurornis »

_zoltan_ wrote: post #7
Aurornis wrote: post #3

Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…

"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.

> Going from paper to implementation from scratch in half an hour or so is great.

This repo isn’t showing that at all. Scroll to the bottom of the README and you’ll see the other project it was based on. It’s a translation of other people’s work.

There have been dozens or perhaps hundreds of vibecoded TurboQuant examples posted around the usual forums in the past few days. This one doesn’t even include anything helpful like benchmarks or tests. It’s just some proof of concept code that doesn’t even work if you try to run it.

My problem with this specific type of vibe coded project is that it’s initially presented as something more novel or polished in order to get more upvotes, karma, likes, or pad a resume. Then you read it and discover they just pointed Claude at some other projects and told it to produce something similar, then posted it as their own work.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#30

Post by aegis_camera »

simonw wrote: post #21

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Downloa…

git clone https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue:

# Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/

Post reply on HN