Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

231–240 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#231

Post by cptskippy »

CamperBob2 wrote: post #223
alexellisuk wrote: post #211

Earlier quoted context omitted.

Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.

How about your reply here? Was that AI-generated? If not, are you conscious of how much you're starting to sound like AI? Is that something you see as a positive thing, or something you'd like to avoid? I actually find this somewhat interesting, because it seems that a lot of people who weren't comfortable with expressing themselves verbally are feeling more empowered in that area. We're hearing new voices for the fi…

It's an interesting idea. Tools like Grammarly exist to help with business communication. I wonder if there's a space for a Social Media or online writing assistant to help people. I for one could probably benefit from a tone shift away from acerbic troll.

Re: Local Qwen isn't a worse Opus, it's a different tool

#232

Post by mistercheese »

I’m not sure if I missed it, but I’m curious how you feel about cloud hosted models with ZDR policies? GLM5.2 or even Minimax M3 on Fireworks or Together ai should be still relatively/consistently cheap and private but a lot more capable and easier to setup?

Re: Local Qwen isn't a worse Opus, it's a different tool

#233

Post by alexellisuk »

mistercheese wrote: post #232

I’m not sure if I missed it, but I’m curious how you feel about cloud hosted models with ZDR policies? GLM5.2 or even Minimax M3 on Fireworks or Together ai should be still relatively/consistently cheap and private but a lot more capable and easier to setup?

Thanks for the comment ZDR is mentioned in the post - in particular many the coding plans that are not from the two major leaders have questionable IP/ownership claims on inputs/outputs :)

And ZDR is still data sharing with a third party. This is the essence of an enterprise agreement, it's not allowed, even if they pinkie promise not to store it.

If your customers allow you to share their data with third parties, then ZDR may be an option for you. I am not a laywer.

Where I see ZDR as being more relevant is in protecting your employer's IP - not allowing a missed setting to mean AI labs can train, retain, and publish/resell your work. It's what we'll consider when the subsidies stop being available - open-router, ZDR - but for coding - not for customer data. Very important distinction.

Re: Local Qwen isn't a worse Opus, it's a different tool

#234

Post by lelandbatey »

theshrike79 wrote: post #43
sanderjd wrote: post #15

Earlier quoted context omitted.

Right. Opus 4.5 8 months ago, good enough for agentic coding. How far behind that are open weight models? More than 8 months? But how much more? When will they reach Opus 4.5 level? A few months from now? A year from now? Never?

The power of Opus isn't just the model, it's in the harness too. You can try it by using Opus through Github Copilot vs official Anthropic tools. You'll get very different results and experience (in my opinion).

I've tried Opus 4.6 in the Opencode harness through the Github Copilot API, and I've tried Opus 4.8 in Claude Code. I found I preferred Opus 4.6 in Opencode (and in general, I like Opencode much more in that it hid less from me). I found both to be pretty similar as far as efficacy (I was surprised that Opus 4.8 felt like such a minor improvement over 4.6).

Re: Local Qwen isn't a worse Opus, it's a different tool

#235

Post by lelandbatey »

alexellisuk wrote: post #215
barrkel wrote: post #67

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single…

vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage) Dismissed is a strong term, but let me give you some more details. It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower. And the worst bit? With all that work - setting it up and t…

Bashing on ollama is totally warranted, since ollama is a UI skin around llama.cpp and that's it. If all you cared about was "I want to run a model and use it via an API" then the only thing it did was give you a GUI to download models (vs browsing HuggingFace yourself and downloading .gguf files yourself) and a GUI with a button labeled "run" (instead of a run.sh or run.bat script launching llama-server).

That's not _nothing_, but it's pretty close to nothing, and for the prosumer crowd it edges towards "just gets in the way".

Re: Local Qwen isn't a worse Opus, it's a different tool

#237

Post by LogicFailsMe »

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I find with Claude that when I call its BS I get better results. And it openly admits to lying to and gaslighting me as well as not seeing any way to stop itself from continuing to do so.

Fable seemed less apt to do so but I didn't get enough time with it before it was yanked away to know for sure. It may have had mixed results on the benchmarks but it was finding bugs opus never found.

Re: Local Qwen isn't a worse Opus, it's a different tool

#238

Post by rs38 »

ttsiodras wrote: post #214
alexellisuk wrote: post #212

Earlier quoted context omitted.

We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp…

I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard). But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability. Regardless - thanks for putting the eff…

wanna chime in, recently tried vLLM to consume a NVFP4 Gemma4 safetensor model and see how the batching can show up in nice t/s numbers. it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up. Appreciate how llama.cpp simply works and does things fast and obvious

Re: Local Qwen isn't a worse Opus, it's a different tool

#239

Post by theshrike79 »

sanderjd wrote: post #180
theshrike79 wrote: post #41

Earlier quoted context omitted.

We shouldn't just measure the power of the raw LLM, harnesses matter more and more. It's like taking the engine out a each car, putting it to a test bed and running it and then making a decision whether the car is good or bad based on the graphs the test bed provided. You might have the best engine in the world, but if you put it in a shit car, the result is still bad. The seats are squeaky plastic, the infotainment…

Aren't there benchmarks that measure at the harness level as well?

How would you benchmark "agent harness communicates with user clearly" it's 100% a feels measurement.

Re: Local Qwen isn't a worse Opus, it's a different tool

#240

Post by Forgeties79 »

headcanon wrote: post #192
Forgeties79 wrote: post #84

Earlier quoted context omitted.

Musical instruments are not random. You’re just doing random inputs. Instruments are consistent, even if the “flavor” and quality varies with different builds. Playing a B on a saxophone always plays a B.

I see you haven't tried a modular synthesizer yet :) Getting back to the same "place" in a patch can sometimes be impossible, and it does feel "random" until you get the hang of it.

But ultimately it isn’t unpredictable and random. That’s just a skill issue. There is literally no person good enough at prompting to create consistent, predictable, useful results.
Post reply on HN