Another project without running real benchmarks. It's very easy to generate tokens, it's much harder to solve tasks locally.
Here is a reference https://www.sharpai.org/benchmark/
For specific tasks, local model could achieve workable level.
TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
41–50 of 54 posts
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#42[dead]
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#43Feels 100% vibe coded in a bad way. Llama.cpp already has KV compression and one of the turbo quant PRs will get merged at some point. If you don’t care about the fancy 3 bit, the q8 KV compression is good enough! Don’t bother with q4 ./build/bin/llama-server -m model.gguf \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -c 65536 Etc
One of my user requested MLX comparison with GGUF, he wanted to run the benchmark, I was thinking about how to get MLX support without bundling the python code together with SharpAI Aegis, a Local or BYOK local security agent https://www.sharpai.org.
Then I had to pick up the Swift and create it.
The benchmark shows a benefit of MLX engine, so it's user's choice which engine to use, aegis-ai supports both : )
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#44Earlier quoted context omitted.
"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.
Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…
> without making clear how much effort has gone into it
I'm increasingly convinced this is the critical context for sharing LLM outputs with other people. The robots can inflate any old thought into dozens of pages of docs, thousands of lines of MR. That might be great! But it completely severs the connection between the form of a work and the author's assessment/investment/attachment/belief in it. That's something one's audience might like to know!
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#45Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by p…
The performance gain in the recent Flash-MoE implementations is seemingly obtained mostly by coalescing the data for each single MoE layer-expert into a single sequential extent which can be read efficiently from SSD. If so, this will actually require some changes in the underlying GGUF format; though the GGUF standard provides explicitly for specifying different data layouts, so the additions are arguably minor.
As far as the TurboQuant thing goes, it seems that attn-rot has recently been merged in, which is a lightweight variety of it and written by the original llama.cpp author, so not an outside pull req.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#46Earlier quoted context omitted.
"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.
Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…
Is t the point of an MVP to be an MVP?
The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab.
The purist tests being asked for is in contradiction to the ShowHN guidelines.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#47Earlier quoted context omitted.
Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…
Is t the point of an MVP to be an MVP? The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab. The purist tests being asked for is in contradiction to the ShowHN guidelines.
[dead]
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#48Earlier quoted context omitted.
Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…
Is t the point of an MVP to be an MVP? The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab. The purist tests being asked for is in contradiction to the ShowHN guidelines.
This post wasn't marked as a Show HN.
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#49Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there.. Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach. Am i missing something fundamental?
llm intelligence seems to be proportional to the ram used. All techniques like this will be used by everyone.
You can almost always use less RAM by making inference slower. Streaming MoE active weights from SSD is an especially effective variety of this, but even with a large dense model, you could run inference on a layer-wise basis (perhaps coalescing only a few layers at a time) if the model on its own is too large for your RAM. You need to store the KV-cache, but that takes only modest space and at least for ordinary transformers (no linear attention tricks) is append-only, which fits well with writing it to SSD (AIUI, this is also how "cached" prompts/conversations work under the hood).
Re: TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
#50Earlier quoted context omitted.
Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it. Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely…
Is t the point of an MVP to be an MVP? The OP put together a POC and shared it, showing novel concepts used together. They are not some large R&D lab. The purist tests being asked for is in contradiction to the ShowHN guidelines.
> The OP put together a POC and shared it, showing novel concepts used together.
That's the contention: There are countless POCs for these concepts already, and some of them were used as the basis for this project.
It's not really a novel POC, it's the result of putting the previous work into Claude Code and telling it to rewrite it in Swift, then putting your name on it. To be fair, the person did start adding the reference projects to the very end of the README
But if you didn't what to look for, you'd assume this was a very novel project attributable to their own work