71–80 of 138 posts
Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?
I would be very interested in a similar benchmark for *KV cache* quantizations. I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there. I also heard that KV cache quantization matters more with longer contexts.…
Earlier quoted context omitted.
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
Beats Opus 4.6 at what exactly? It certainly isn't code. I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...
Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.
If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.
I've been running Unsloth IQ3_S on my 5060ti with mmproj offloaded, getting 600-1000 prefill and 30-50 tg with this config: /data/llm/llama.cpp/build/bin/llama-server --threads 4 --threads-batch 8 --batch-size 4096 --ubatch-size 256 --port 9999 --temp "1.0" --top-p "0.95" --top-k "20" --min-p "0.0" --presence-penalty "0.0" --reasoning auto --reasoning-preserve --reasoning-budget 4096 --gpu-layers-draft all --spec-typ…
The chat template is froggeric's fixed qwen template, v22.5 as of today.
> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold. Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better . I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?
On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…
Yah, prediction interval instead, right? (I'm still learning statistics) Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
Earlier quoted context omitted.
Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?
Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.
Earlier quoted context omitted.
Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.
I normally don't appreciate snarky comments on HN, me being but a simple curmudgeon wanting pure information and the occasional opinion. but this one actually makes a good point, even if it's not stated explicitly. Reading this book is a wonderful experience because of the way it's written. Reading the plot summary (from an AI or otherwise) yields nothing but the dry husk of that experience.
Or they don't admit to reading fiction...