291–300 of 317 posts
I really like what Qwen are doing, and a lot of these Chinese labs, but until I can ask their models what happened during the student protests in 1989 or why human rights groups are upset about the Uighurs and the model gives me a straight answer I'm just not able to trust these models with anything of substance.
Earlier quoted context omitted.
personally seen a lot of people switch to Kimi and Qwen after Opus 4.7. Kimi 2.6 feels like Opus 4.6 which, to me, was a great model for 98% of coding tasks
Frontier: Need it done quick and I'm willing to pay. Open-weight: Good enough for the majority of tasks, and I'm willing to spend a bit more time and effort steering towards my desired result.
95% of the work most of us do is mostly just plumbing - connecting X and Y together. A ton of grunt work - writing basic loops, fetch statements, importing libraries. You really don't need PhD level intelligence to handle these
The only time you need Opus 4.7+ tier intelligence is when you're quashing a nasty bug or refactoring something complex
Earlier quoted context omitted.
These are dynamic quants, and they're basically just an indication of how far away from the desired quant it is allowed to go to achieve the goal. Generally, unsloth's toolchain moves quants up, rarely down. * _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block. * K quants do…
Hey some of us are on hardware (gfx906 based Radeon MI50s with 32GB of stupidly fast VRAM and basically no compute) that inference significantly faster with Q_0 and Q_1 quants
Its not amazing at compute (yet is a member of the GCN family, which I have been a fan of since its inception) and ended up being too expensive for perf/$ and perf/watt.
The only thing it did was make Nvidia rush Series 10 out the door and make it too good. Nvidia has been unable to live up to the gen-to-gen uplift Series 10 did, all because AMD made Nvidia blink.
Basically, you're 2 gens too early. CDNA2/gfx90a is the minimum you need to get any meaningful performance out of inference, or maybe CDNA1/gfx908 if you really don't need to quantize at all.
BTW, I did suggest this elsewhere in this HN story, but have you tried just disabling KV quant entirely? That is a huge speed uplift for compute-poor users.
Also, llama.cpp's support for gfx906 is probably never going to as good as it is for other cards, and good ROCm support for cards before they rebooted the driver/stack team is probably never going to materialize. I don't see the point in hanging onto them.
Like, if I was in your place, replacing it with even a 9060xt, with half the RAM, would be a step up. They go for $450. People have been building dedicated inference machines with these and they've been amazing, just throwing in 3 or 4 in, and scaling VRAM to meet needs.
Earlier quoted context omitted.
I'd held off from buying a new personal laptop for quite a few years and felt that the M5-128gb was justifiable once I started really seeing payoffs from using AI at work. Running w/ Cursor and doing some "nights and weekends" type coding / conversations, I was hitting $100-200 of usage within a few weeks. I know there's probably better ways to manage costs, but I was getting enough value out of it to keep bumping my…
Thank you! That was a generous and helpful response, I really appreciate it. Food for thought... > "...if you're strategic about your usage, the ability to "talk for free" and occasionally "burst" to an online provider or having some hugging-face tokens to try out different models that you can't quite run locally is really nice. Talking to the AI (locally) to even just do non-coding planning without worrying about da…
Again: I'm finding waaaay enough utility that I'm tempted to invest more "CapEx" and get a used system for day-to-day, "always on" local work... but more literally, that's probably a better job for "OpEx"! Tune my "crontab" work against local models and then max out at a $1/day budget slaved to an always on RPI connected to ethernet at home.
$365/year of off-site AI lasts 10 years before I come close to recouping the hardware (and electricity) costs of having "yet another device" purchased and turned on 24x7... and certainly there will come a day when you go to the store and buy a $200-500 "TITO" device (Tokens In => Tokens Out) that plugs into a ~30-60W USB-C port before then.
If you're using HF tokens (or "rent-a-A100" or whatever), are always connected to home ethernet (Sun Microsystems: The Network IS the Computer), and maybe supplement with a Kagi backend for attaching to the raw internet then you get _most_ of the surety of "my queries are private" unless you're locally hacked or are the target of nation-state scrutiny. :shrug:?
Keep in touch if you end up doing something cool with all this! $USERNAME@yahoo.com (and hopefully I'll have my AI setup filtering out all the viagra spam before then!).
Earlier quoted context omitted.
Nobody releases numbers that show them to be worse than competitors lol. This even applies to OpenAI & Anthropic who don't even eval on the same datasets a lot of the time.
I do recall mistral doing this. It's not always about being the best, but also fastest or smallest. The user should have all the information for its own use case.
Which is fine, we all have to make money, but it is disingenuous. It's just unfortunate that running some of these benchmarks is so expensive that it's not really realistic for most companies to actually run them.
Earlier quoted context omitted.
here's a simple setup to get you started on an Apple M1 Max from 2021 with 32GB VRAM. it will download 20GB of models to `~/.cache/huggingface/hub`, which you can delete when you're done. /Users/gcr/llama.cpp/build/bin/llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M --no-mmproj-offload --fit on -c 65536 # edit to taste --reasoning on --chat-template-kwargs '{"preserve_thinking": true}' --sleep-idle-seconds 90 #…
Can you elaborate more on the differences in running ollama or lmstudio? Do they actually slow down the speed of the inference and if so why? Or is it just a preference thing?
LM-Studio doesn't support certain parameter combinations. For instance, LM-Studio supports KV quantization....but if you're using the MLX backend, you can't set the context length when KV quantization is used? Why? Running a model with certain settings requires keeping a little SAT solver going in your head. I found that overwhelming, so I just stopped using it.
The Ollama devs want to offer a central curated experience, but I perceive their approach as "playing fast and loose." They've re-implemented unique code for every model they support in their own Go runtime, so certain parameter choices aren't supported. On my hardware, their MLX backend just doesn't work at all without segfaulting the server process for example. It doesn't smack as vibe coded the way oMLX does, but it also doesn't smack as professional or battle-tested.
Ultimately, just dropping down to llama-cpp's GGUF model support and asking for default settings has provided faster inference speeds than anything I've been able to benchmark with them, but everything's within 10% of each other anyway so it's not a huge deal for me.
I was getting dangerously close to my weekly Claude Code limit last night so I had Claude set up Qwen3.6 with llama.cpp and OpenCode. Honestly it's a great (free!) alternative to Claude Code--certainly more than good enough for a lot of smaller less complex tasks. I'm excited to try this new version. The fact that open-source models are so close to the frontier is very impressive.