Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local Qwen isn't a worse Opus, it's a different tool

blog.alexellis.io

Post reply on HN
Print view View original

111–120 of 271 posts

Re: Local Qwen isn't a worse Opus, it's a different tool

#111

Post by ACCount37 »

coldtea wrote: post #103
dkersten wrote: post #80

Earlier quoted context omitted.

> This is not scientific at all, just vibes, YMMV. This is the problem. I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X" , or "model Y should be used in ways Z" . But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive,…

> I would love to have a product sheet showing what each models strengths an weaknesses are, so that I can have a clear decision tree of "if this kind of work, use model X", or "model Y should be used in ways Z". But they all look the same from the outside and the only way to figure out which might be marginally better at what is to do extensive, time consuming, and perhaps expensive testing. Think of it less like a…

One issue with that is that human helpers last longer. LLMs cycle in and out in months, and what held for Your Favorite LLM 6.7 may not hold for Your Favorite LLM 6.9.

Re: Local Qwen isn't a worse Opus, it's a different tool

#112

Post by trey-jones »

throw310822 wrote: post #55
i_idiot wrote: post #5

Earlier quoted context omitted.

> Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. They really are fantastic for a lot of use cases and I think most people do not need SOTA. When I run that qwen model in my measly 4070 12 GB for my personal email agent that I build and experiment with, I need privacy more than anything else. It does a great job. Even for coding tasks, given you know how to use them inst…

> I think most people do not need SOTA SOTA can code but can also prove theorems and teach you about music theory or ancient Greece's substrate language or botany. Speaking in tens of different languages. I wonder how many hundreds of billions of parameters can be saved just by removing much of the general knowledge parts while keeping logical and programming abilities the exact same.

Exactly. I have sort of a fetish for trying to make things smaller by trimming out things that aren't needed. Unfortunately this skill has been largely useless since forever, because hardware improves to the point that these optimizations are trivial:

Network Bandwidth, Storage space and speed, memory capacity. While all of these were worth optimizing for at a point in history, that point is behind us. It's probably a reasonable expectation that it will eventually be true for VRAM.

Re: Local Qwen isn't a worse Opus, it's a different tool

#113

Post by krzyk »

theplumber wrote: post #19
sanderjd wrote: post #15

Earlier quoted context omitted.

Right. Opus 4.5 8 months ago, good enough for agentic coding. How far behind that are open weight models? More than 8 months? But how much more? When will they reach Opus 4.5 level? A few months from now? A year from now? Never?

I think in the next 6 months we will have Opus 4.5 performance in open models. We are very close

We need first to reach level of Sonnet 4.x, we aren't at that level yet.

Re: Local Qwen isn't a worse Opus, it's a different tool

#115

Post by krzyk »

barrkel wrote: post #67

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single…

AFAIR the general consensus is (was?): - llama.cpp for single user - vLLM for multi-user (e.g. enterprises)

They are similar, but for different use cases.

Re: Local Qwen isn't a worse Opus, it's a different tool

#117

Post by trey-jones »

asimovDev wrote: post #78
whazor wrote: post #57

Would be interesting to use local models for: - tool calling - code base exploration - anonymizing / abstracting your request Such that your local AI communicates to frontier model like an expensive consultant giving high level advice. I think due to the lower latency of a local model that this could be faster.

I used Qwen 27b 8 bit MLX version on a decompiled android APK recently. It succesfully identified how it worked even the obfuscated classes and methods. It wrote a 1000 line documentation with examples but the time was dreadful. At some point it slowed down to 5 t/s so the whole thing took over an hour , the writing of documentation alone was over 40 minutes, fans blasting the entire time.

I know it uses electricity, but part of the benefit of a local model has to be that you can let it do this while you sleep, and not pay Anthropic for an unknown number of tokens.

Re: Local Qwen isn't a worse Opus, it's a different tool

#118

Post by asimovDev »

trey-jones wrote: post #117
asimovDev wrote: post #78

Earlier quoted context omitted.

I used Qwen 27b 8 bit MLX version on a decompiled android APK recently. It succesfully identified how it worked even the obfuscated classes and methods. It wrote a 1000 line documentation with examples but the time was dreadful. At some point it slowed down to 5 t/s so the whole thing took over an hour , the writing of documentation alone was over 40 minutes, fans blasting the entire time.

I know it uses electricity, but part of the benefit of a local model has to be that you can let it do this while you sleep, and not pay Anthropic for an unknown number of tokens.

yeah i totally understand and I am thoroughly impressed it works. And the electricity cost isn't that bad since it was on a ARM laptop (MacBook M3 Max) and not a beefy workstation with a GPU. I just let the agent do its work while watching the World Cup.

Re: Local Qwen isn't a worse Opus, it's a different tool

#120

Post by bandrami »

glerk wrote: post #6

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative.…

I think this goes beyond "vibes" to cargo-culting. It's why nobody's ever able to actually show ROI from LLMs
Post reply on HN