251–260 of 848 posts
I wonder if Anthropic and OpenAI possibly missed the window to go public. A 27B open-weight model trading blows with the SOTA from just half a year ago is not great news for trillion-dollar investments...
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…
Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop? If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU. How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Here's my config: https://gitea.va.reichard.io/evan/nix/src/branch/master/modu...
Earlier quoted context omitted.
Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?
We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
Earlier quoted context omitted.
> Beats Opus 4.7 Max I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage. Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expecta…
"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.
Qwen 3.6 is ~$2/m tok, 3.8 should be drop in replacement. Gemma 31B is $0.34/m tok. The price differential on these models is massive on openrouter.
Yeah, I would appreciate if someone could make sense of the pricing differences between these models. How can a provider run DSv4F at lower cost than a 27B dense or 35B A3B model? Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)? DeepInfra prices: Qwen 3.6 27B: $0.32 in / $3.20 out Gemma 3 27B: $0.08 in / $0.16 out DeepSeek V4 Flash 0731: $0.08 in / $0.18 out…
The file "Just loads" on llama.cpp, the Unsloth https://huggingface.co/unsloth/Qwen3.8-27B-GGUF is an MTP file, I see mostly the same speed on pp and generation. There has to be something wrong with those benchmarks, I find extremely hard to believe a 27B model can work similar or exceed opus 4.6.
To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.
There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.
Earlier quoted context omitted.
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort. put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, tas…
You sound like your trying to reassure yourself of something. I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
Most managers do though?