TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
1–10 of 54 posts
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#2We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro:
TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead.
SSD Expert Streaming: To fit a 122B parameter model (e.g., Qwen3.5-122B MoE) without triggering macOS VM swapping or Watchdog kernel kills, the full ~60 GB weight file remains on NVMe. Only the top-k active expert pages are streamed to the GPU per forward pass at ~9 GB/s. As a result, inference runs with only 2,694 MB of active GPU VRAM on the M5 Pro 64GB, while the OS page cache automatically handles hot-expert reuse.
By combining these two approaches, we can comfortably run massive models in memory-constrained environments on Apple Silicon.
Also tested QWEN 4B on IPHONE 13 Pro.
Code and implementation details: https://github.com/SharpAI/SwiftLM
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#3Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit.
At least this one gave credit to the upstream projects which it used as a reference.
The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by pointing claude at the repo and the original paper and having it produce something.
Almost none of these attempts contain information that really matters, like actual benchmark tests with differen KV quantization levels (not just perplexity or KLD).
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#4Does this use anything from the flash-moe project?
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#5We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To…
what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#6I like this idea on expert streaming. I've been poking around fairly thoroughly at the same idea - can we fix a set of experts? when can we fix them? How long is the top-k selection "good" for in terms of number of forward passes?
One thing I've turned up in smaller models and I'm sort of winding my way toward verifying in larger ones is that if you train the MoE model from scratch with this kind of knockout / subset of experts baked in, then you get significantly better loss outcomes. In small models, it's actually better than training an MOE without conditioning on a reduced set of experts per pass.
Anyway, pretty cool. There's some Pareto-optimal curve based on memory bandwidth, amount of GPU / unified RAM and inference compute times for streaming stuff in.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#7Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…
"vibe coded" is NOT the bad thing you think it is.
Going from paper to implementation from scratch in half an hour or so is great.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#8Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…
"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.
> "vibe coded" is NOT the bad thing you think it is.
It's not inherently bad in the same way that a first draft of a novel is not inherently bad.
But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#9Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…
"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.
That’s a starting spot, but how about some testing and benchmarks?
Where’s the value added if the person just tells Claude to do it and then submits a PR?
The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#10We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To…
what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md
yeah this I'd like to see added to teh readme.