Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

221–230 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#221

Post by cptskippy »

askvictor wrote: post #22
cptskippy wrote: post #4

I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and…

Does Intel make decent GPUs now? I must be out of the loop...

I'm using an Intel Arc Pro B70 which has 32 GB of VRAM. It's estimated to get ~35-45 t/s at $21-27 $/t/s. An RTX 5090 is ~61 t/s at ~$33 $/t/s.

So in terms of raw power Nvidia is effortlessly still king, but in price-to-capacity Intel is best in class.

Intel's Battlemage GPUs also natively support SR-IOV and GPU partitioning which allows you to isolate workloads. This is useful in homelab environments if you have workloads that benefit from GPU acceleration. I was able to split the B70 into 4 virtual GPUs and hand them out to Frigate NVR, Plex, and other workloads.

Re: Local Qwen isn't a worse Opus, it's a different tool

#222

Post by cptskippy »

Ritewut wrote: post #20
jauntywundrkind wrote: post #17

Earlier quoted context omitted.

What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.

Tokens per second. The difference between 8B and something like 16B is not as big as you might think in practical usage and 8B is a lot faster and interactive than 16B but there are certain things where it is useful to farm it out to the large model.

Exactly this.

Creating conversation titles and parsing HTML/JSON don't benefit from 27B models.

The B70 can run both models comfortably side-by-side so it makes better use of time and resources.

Re: Local Qwen isn't a worse Opus, it's a different tool

#223

Post by CamperBob2 »

alexellisuk wrote: post #211
skipants wrote: post #179

I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense: > These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols. Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill. And it's obviously fully AI-generated…

Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.

How about your reply here? Was that AI-generated? If not, are you conscious of how much you're starting to sound like AI? Is that something you see as a positive thing, or something you'd like to avoid?

I actually find this somewhat interesting, because it seems that a lot of people who weren't comfortable with expressing themselves verbally are feeling more empowered in that area. We're hearing new voices for the first time, albeit heavily-filtered ones, and I have to believe that's a good thing.

But part of me still finds it offputting for some reason. It's interesting to think about whether that's more of a "you" problem, or more of a "me" problem.

Re: Local Qwen isn't a worse Opus, it's a different tool

#224

Post by QwenGlazer9000 »

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

As someone who actually uses musical instruments, it's not at all the same. If anything, traditional IDEs are closer to musical instruments, which seem to be going EOL if you listen to the hype bros.

Re: Local Qwen isn't a worse Opus, it's a different tool

#225

Post by tingletech »

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I do think it pays to be nice to the model. When the context window is running out I like to ask "please summarize what went well and what didn't work in this session. How could the user be more helpful?"

Re: Local Qwen isn't a worse Opus, it's a different tool

#226

Post by john_strinlai »

tingletech wrote: post #225
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I do think it pays to be nice to the model. When the context window is running out I like to ask "please summarize what went well and what didn't work in this session. How could the user be more helpful?"

>I do think it pays to be nice to the model.

there was something on HN a few weeks ago about how most/all models perform better the more rude you are to them.

(i still say "please", i can't help it)

Re: Local Qwen isn't a worse Opus, it's a different tool

#227

Post by djeastm »

devin wrote: post #220
glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

It is not at all like playing an instrument. Instruments present a clear interface to a user, have predictable outputs, etc. The only comparison that might work for me is that LLMs are very bad instruments where you are constantly forced to negotiate its idiosyncrasies in order to massage the output you want from it, and even then there is enough randomness that trying to do so is almost a fool's errand.

I think they mean playing different instruments not other instances of the same instrument. A tuba's interface differs from a violin's, etc.

Re: Local Qwen isn't a worse Opus, it's a different tool

#228

Post by squidbeak »

mahidhar wrote: post #173
coldtea wrote: post #103

Earlier quoted context omitted.

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

Well, unlike a human, I cannot expect any these LLMs to take any ownership of the work they do. I cannot expect any given model and version (sonnet 4.6) to learn, improve and adapt over time. I cannot expect it's limitations to ever go away at the model level. So it is not like a human in most ways that I actually care about. That said, I can't wait for LLMs to stop being AI and start being just another tool. Anythin…

You're on the hook for their work in the way a manager is for their staff's output. The insistence of AI being a mere tool very often comes with this strange desire to be free of responsibility for its work. People seem to forget that the big advantage in these things is the range they have for obscure insight and creative solutions, both impossible with determinism.

Re: Local Qwen isn't a worse Opus, it's a different tool

#229

Post by devin »

djeastm wrote: post #227
devin wrote: post #220

Earlier quoted context omitted.

It is not at all like playing an instrument. Instruments present a clear interface to a user, have predictable outputs, etc. The only comparison that might work for me is that LLMs are very bad instruments where you are constantly forced to negotiate its idiosyncrasies in order to massage the output you want from it, and even then there is enough randomness that trying to do so is almost a fool's errand.

I think they mean playing different instruments not other instances of the same instrument. A tuba's interface differs from a violin's, etc.

My criticism of the comparison would stand in either case. There is nothing clear and uniform about the interface to LLMs that match their musical counterparts. Even modular synthesizers with random sources are far more controlled.

I also think it's disingenuous to call LLMs "tools" in the stricter sense of the definition, but I've mostly given up trying to convince people of this. Main reason being that a terrible writer and a gifted writer can produce similar outputs, and for the terrible writer it will be above their average, and for the gifted writer it will be below what they could produce with full control.

Re: Local Qwen isn't a worse Opus, it's a different tool

#230

Post by alentred »

alexellisuk wrote: post #211
skipants wrote: post #179

I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense: > These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols. Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill. And it's obviously fully AI-generated…

Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.

> Fair enough ... compressed ... ACTION->RESULT ... NEGATION->STATEMENT ... follow up questions.

What model are you again?

Post reply on HN