31–40 of 286 posts
What kind of hardware does HN recommend or like to run these models?
What kind of hardware does HN recommend or like to run these models?
Smells like hyperbole. A lot of people making such claims don’t seem to have continued real world experience with these models or seem to have very weird standards for what they consider usable. Up until relatively recently, while people had already long been making these claims, it came with the asterisks of „oh, but you can’t practically use more than a few K tokens of context“.
Is this actually true? I want to see actual evals that match this up with Sonnet 4.5.
Obviously there's more to a model than that but it's a data point.
[1]: https://github.com/fairydreaming/lineage-bench
[2]: https://github.com/fairydreaming/lineage-bench-results/tree/...
Earlier quoted context omitted.
> Somewhere between Haiku 4.5 and Sonnet 4.5 That's like saying "somewhere between Eliza and Haiku 4.5". Haiku is not even a so-called 'reasoning model'.¹ ¹ To preempt the easily-offended, this is what the latest Opus 4.6 in today's Claude Code update says: "Claude Haiku 4.5 is not a reasoning model — it's optimized for speed and cost efficiency. It's the fastest model in the Claude family, good for quick, straightfo…
Haiku 4.5 is a reasoning model. [0] [0]: https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c3630... > Claude Haiku 4.5, a new hybrid reasoning large language model from Anthropic in our small, fast model class. > As with each model released by Anthropic beginning with Claude Sonnet 3.7, Claude Haiku 4.5 is a hybrid reasoning model. This means that by default the model will answer a query rapidly, but users have…
I would absolutely believe mar-ticles that Qwen has achieved Haiku 4.5 'extended thinking' levels of coding prowess.
Quite misleading, really.
Earlier quoted context omitted.
It's less than you'd think. I'm using the 35B-A3B model on an A5000, which is something like a slightly faster 3080 with 24GB VRAM. I'm able to fit the entire Q4 model in memory with 128K context (and I think I would probably be able to do 256K since I still have like 4GB of VRAM free). The prompt processing is something like 1K tokens/second and generates around 100 tokens/second. Plenty fast for agentic use via Ope…
I've had an AMD card for the last 5 years, so I kinda just tuned out of local LLM releases because AMD seemed to abandon rocm for my card (6900xt) - Is AMD capable of anything these days?
What kind of hardware does HN recommend or like to run these models?
I think the 27B dense model at full precision and 122B MoE at 4- or 6-bit quantization are legitimate killer apps for the 96 GB RTX 6000 Pro Blackwell, if the budget supports it. I imagine any 24 GB card can run the lower quants at a reasonable rate, though, and those are still very good models. Big fan of Qwen 3.5. It actually delivers on some of the hype that the previous wave of open models never lived up to.
What kind of hardware does HN recommend or like to run these models?
The cheapest option is two 3060 12G cards. You'll be able to fit the Q4 of the 27B or 35B with an okay context window. If you want to spend twice as much for more speed, get a 3090/4090/5090. If you want long context, get two of them. If you have enough spare cash to buy a car, get an RTX Ada with 96G VRAM.