TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS • Hacker News Board

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#51

Post by Aurornis » Wed, Apr 01, 2026, 9:52 PM UTC

zozbot234 wrote: post #45

Aurornis wrote: post #3

Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…

The performance gain in the recent Flash-MoE implementations is seemingly obtained mostly by coalescing the data for each single MoE layer-expert into a single sequential extent which can be read efficiently from SSD. If so, this will actually require some changes in the underlying GGUF format; though the GGUF standard provides explicitly for specifying different data layouts, so the additions are arguably minor. As…

> As far as the TurboQuant thing goes, it seems that attn-rot has recently been merged in, which is a lightweight variety of it and written by the original llama.cpp author, so not an outside pull req.

Yes, read the first sentence of the PR for it. The project is a constant target for vibecoded PRs and they're trying to stay in front of it:

> In anticipation of the incoming flood of vibe generated PRs implementing TurboQuant, I'm raising the baseline a bit

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#52

Post by aegis_camera » Wed, Apr 01, 2026, 10:10 PM UTC

simonw wrote: post #48

dalemhurley wrote: post #46

Earlier quoted context omitted.

Is t the point of an MVP to be an MVP? The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab. The purist tests being asked for is in contradiction to the ShowHN guidelines.

This post wasn't marked as a Show HN.

Tried, but wrong time to post, it got zero attention . :)

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#53

Post by aegis_camera » Wed, Apr 01, 2026, 10:17 PM UTC

dalemhurley wrote: post #46

simonw wrote: post #19

Earlier quoted context omitted.

Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…

Is t the point of an MVP to be an MVP? The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab. The purist tests being asked for is in contradiction to the ShowHN guidelines.

Thanks, we are not large R&D lab, limited resources. We were working on a product with is a Local VLM first BYOD when you want Video Security application, our users requested to have a MLX backend benchmark comparison, we tried hard to not deliver with Python in the application bundle, so we searched for a pure binary based MLX implementation the results shown we need to build one. It took us two weeks to get it working and we had been testing with multiple models. As a reference, you can see the result here: https://www.sharpai.org/benchmark/

Then we saw the announcement from Google about TurboQuant, it's so cool, so we started to integrate them (along with SSD/Flash streaming). It's a non-trivial process and thanks for your support and understanding. When we saw the mobile application alive with QWEN 3 1.7B model, we thought it worth.

If we get anything similar with well maintains, we will definitely adopt it since our target is the production delivery, if this one gets good support from the community, we will continue to support.

I think all the posts here gave us a reason to continue.

Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

#54

Post by brokencode » Thu, Apr 02, 2026, 7:24 PM UTC

yieldcrv wrote: post #11

brokencode wrote: post #9

Earlier quoted context omitted.

That’s a starting spot, but how about some testing and benchmarks? Where’s the value added if the person just tells Claude to do it and then submits a PR? The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

if it works it works we live in a wholly unoptimized world because the available resources have been so high, while the benefits of optimizing have been so low. that has flipped now and there are tons of low hanging fruit to optimize. I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

Who says it works if the “author” isn’t thoroughly testing and reviewing it?

People who do this want the fun part of pretending they’re implementing a feature without actually putting in the hard work it takes to make something for real.

They want the repo maintainers to do all the hard, boring parts while they have fun. As if maintainers of open source projects don’t have enough thankless work on their plates. Good luck with that!