131–135 of 135 posts
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…
Measuring quality e2e definitely makes sense. But I think there is a bit more to this: > Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbe…
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration. On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same. So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
Earlier quoted context omitted.
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration. On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same. So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
Yeah, makes sense; e2e is different and valuable, KLD is not a replacement. As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.
Earlier quoted context omitted.
Your first link is just the pedantry. The second link... I don't see any issue for the question I asked? The confidence interval is supposed to be wrong a certain percent of the time. It being so wrong it disproves itself is funny but the 90% is still 90% isn't it? They just landed in a particularly harsh part of the 10%. The conclusion they come to is "it is possible to do better in the individual case by taking int…
If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”). But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to mak…
Measuring quality e2e definitely makes sense. But I think there is a bit more to this: > Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbe…
> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing: Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baselin…
Way better than wikitext-- but tells you nothing about errors tending to compound or cancel out.
Like say a test shows that only one token in a 10,000 token test would be different. Sounds very close, ship it!-- but what if trajectories with that single different token guarantees failure because it sets in motion a cascade of differences that ultimately result in a final distribution that doesn't include the solution?