Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Find the best local LLM for your hardware, ranked by benchmarks

github.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#21

Post by andyyyy64 »

llagerlof wrote: post #5

What’s new regarding llmfit? https://github.com/AlexsJones/llmfit

Fair question. llmfit answers "will this model fit in my memory?" — it's a fit/size calculator, and a good one. whichllm answers a different question: "of the models that fit, which is actually best?" It pulls candidates, then ranks them by merged real benchmarks (LiveBench / Artificial Analysis / Aider / Arena ELO / Open LLM Leaderboard) with a recency penalty, so a newer 27B beats an older 32B even though both fit — on a 24GB card it puts Qwen3.6-27B above Qwen3-32B on benchmarks, not size.

If "biggest that fits" is the answer you want, llmfit is the simpler tool and Python won't matter to you. If you want "which fitting model is worth running," that ranking layer is the whole reason whichllm exists. Different jobs — I'd genuinely send fit-only users to llmfit.

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#22

Post by andai »

Has anyone gotten the old gpt-oss models running? They scored very high on benchmarks but I constantly had strange problems with them.

So two questions there:

(1) is it actually possible to get good results with them (some people said they got good results, which implies that it might have been hard to get them running properly, but if you can, then they're actually good?). Which also implies the second question,

(2) are benchmarks a spook?

---

...Also, is OP Claude?

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#23

Post by andyyyy64 »

Jasssss wrote: post #2

The plan command is clever. How do you handle the VRAM estimation for models with sliding window attention vs full context? Something like Mistral at 32k context uses way less KV cache than Llama at the same context length, but from the README it looks like the estimation is based on a fixed context size. Does it account for that?

Good catch that's a real gap. The KV estimate is GQA/MQA-aware (per-model head config) but currently assumes dense full-context attention; it does not model sliding-window / chunked attention, so for SWA models like Mistral or Gemma at long context it over-estimates KV. The error is conservative — it tells you a model needs more than it does, not less, so it won't push you into an OOM — but it's still wrong. I'll open a tracking issue with per-architecture window sizes; if you have a reference for the exact SWA configs you care about it'll speed the fix. This is the kind of report I posted for.

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#24

Post by andyyyy64 »

pbronez wrote: post #12

Cool, but it looks like it doesn’t actually test anything on your machine? It does hardware detection and then some lookups. Maybe I missed it but I really want a tool like this to actually run a model on my machine to get the speed numbers. I’ve been using RapidMLX for this. The integrated speed tests matter because the quality of the backend is a moving target and the quantization / MLX format conversion also matte…

You're right that it doesn't run anything — it's a pre-download / pre-purchase decision tool, so it estimates rather than measures by design (you can simulate a GPU you don't own with --gpu). That's a genuine limitation vs running the model: a measured t/s on your exact backend/quant will always beat my estimate. The estimate is bandwidth-bound, per-quant and per-backend, and deliberately conservative on VRAM (weights + GQA-aware KV + activation) so it errs toward "won't fit" rather than crashing you mid-run. Where I can get real measurements I fold them in — calibration data / PRs for specific hardware are very welcome; that's the path to numbers you can trust rather than just plausible ones. On-device measurers like RapidMLX are complementary, a different point in the workflow.

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#26

Post by iknowstuff »

andai wrote: post #22

Has anyone gotten the old gpt-oss models running? They scored very high on benchmarks but I constantly had strange problems with them. So two questions there: (1) is it actually possible to get good results with them (some people said they got good results, which implies that it might have been hard to get them running properly, but if you can, then they're actually good?). Which also implies the second question, (2)…

I had it running on my 128gb strix halo - it ran around 40 tokens per second I think but I found it to be obnoxiously lobotomized.

An uncensored qwen3.5/3.6 is more fun

Re: Show HN: Find the best local LLM for your hardware, ranked by benchmarks

#28

Post by karmakaze »

Not perfect, but I find the artificialanalysis.ai "Intelligence vs. Output Tokens Used in Artificial Analysis Intelligence Index" chart[0] (scroll down to the titled chart) to be of great use. A proper evaluation needs to compare 3 things together: score, speed, and verbosity. This chart plots score vs verbosity.

[0] https://artificialanalysis.ai/?models=gpt-oss-120b%2Cgemma-4...

Post reply on HN