131–140 of 141 posts
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…
Measuring quality e2e definitely makes sense. But I think there is a bit more to this: > Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbe…
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration. On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same. So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
Earlier quoted context omitted.
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration. On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same. So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
Yeah, makes sense; e2e is different and valuable, KLD is not a replacement. As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.
Earlier quoted context omitted.
Your first link is just the pedantry. The second link... I don't see any issue for the question I asked? The confidence interval is supposed to be wrong a certain percent of the time. It being so wrong it disproves itself is funny but the 90% is still 90% isn't it? They just landed in a particularly harsh part of the 10%. The conclusion they come to is "it is possible to do better in the individual case by taking int…
If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”). But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to mak…
Measuring quality e2e definitely makes sense. But I think there is a bit more to this: > Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbe…
> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing: Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baselin…
Way better than wikitext-- but tells you nothing about errors tending to compound or cancel out.
Like say a test shows that only one token in a 10,000 token test would be different. Sounds very close, ship it!-- but what if trajectories with that single different token guarantees failure because it sets in motion a cascade of differences that ultimately result in a final distribution that doesn't include the solution?
Earlier quoted context omitted.
I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.
Depends on the actual audience, I guess. My stated “wisdom” comes in part from /r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side. But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.
That sub is games, porn, and complaints about not having money. A waste of time.
Earlier quoted context omitted.
If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”). But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to mak…
I can see the analogy here but a test like that being binary throws things off and that's also super asymmetrical error. Weren't we sampling a gaussian?
Earlier quoted context omitted.
I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that. So, if there are irrelevant remarks, these are mine. :) Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's init…
Hi, If you ever get to writing a blog post about kv cache quantisation, i'm interested in quantising K differently than V
This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…
Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place. I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.
I have heard good things about Pi which supports auto-compaction, but I can't personally vouch for it since I use my own.
Earlier quoted context omitted.
Your first link is just the pedantry. The second link... I don't see any issue for the question I asked? The confidence interval is supposed to be wrong a certain percent of the time. It being so wrong it disproves itself is funny but the 90% is still 90% isn't it? They just landed in a particularly harsh part of the 10%. The conclusion they come to is "it is possible to do better in the individual case by taking int…
If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”). But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to mak…