Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen2.5-VL-32B: Smarter and Lighter

qwenlm.github.io

Post reply on HN
Print view View original

221–230 of 303 posts

Re: Qwen2.5-VL-32B: Smarter and Lighter

#221

Post by TeMPOraL »

Imustaskforhelp wrote: post #40
jsheard wrote: post #22

Earlier quoted context omitted.

I still don't get where the money for new open source models is going to come from once setting investor dollars on fire is no longer a viable business model. Does anyone seriously expect companies to keep buying and running thousands of ungodly expensive GPUs, plus whatever they spend on human workers to do labelling/tuning, and then giving away the spoils for free, forever?

I think it's market leadership which is just free word of mouth advertising which can then lead to consulting business or maybe they can cheek in some ads in llm directly oh boy you don't know. Also I have seen that once a open source llm is released to public, though you can access it on any website hosting it, most people would still prefer it to be the one which created the model. Deepseek released its revenue mod…

Winner-takes-all markers are never healthy IMO - it's hardly a market when the winner took all.

What I love about "open" models in general and Deepseek in particular, is how they undermine that market. Deepseek drops especially were fun to watch, they were like last minute plot twists, like dropping some antibiotic into a perti dish filled with bacteria. Sorry, try again with a better moat.

"Open" models are in fact the very thing enabling having a functioning market in this space.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#222

Post by samuel »

i_love_retros wrote: post #109

Any security risks running these Chinese LLMs on my local computer?

It's an interesting question! In my opinion, if you don't use tools it's very unlikely it can do any harm. I doubt the model files can be engineered to overflow llama.cpp or ollama, or cause any other damage, directly.

But if you use tools, for example for extending its knowledge through web searches, it could be used to exfiltrate information. It could do it by visiting some specially crafted url's to leak parts of your prompts (this includes the contents of documents added to them with RAG).

If given an interpreter, even if sandboxed, could try to do some kind of sabotage or "call home" with locally gathered information, obviously disguised as safe "regular" code.

It's unlikely that a current model that is runnable in "domestic" hardware could have those capabilities, but in the future these concerns will be more relevant.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#223

Post by WithinReason »

jschoe wrote: post #152
simonw wrote: post #4

Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/

And it still can't answer this: Q: "9.11 and 9.9, which one is larger?" A: "To determine which number is larger between 9.11 and 9.9, let's compare them step by step. Both numbers have the same whole number part: 9. 9.11 has a 1 in the tenths place. 9.9 has a 9 in the tenths place. Since 9 (from 9.9) is greater than 1 (from 9.11), we can conclude that 9.9 is larger than 9.11." "Final Answer: 9.9" I don't think anythi…

You just failed the Turing test, now we know you're an LLM.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#224

Post by int_19h »

simonw wrote: post #6

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

I don't think there's any local model other than full-sized DeepSeek (not distillations!) that is on the level of the original GPT-4, at least not in reasoning tasks. Scoreboards lie.

That aside, QwQ-32 is amazingly smart for its size.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#225

Post by tosh »

YetAnotherNick wrote: post #13
simonw wrote: post #6

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

I don't think these models are GPT-4 level. Yes they seem to be on benchmarks, but it has been known that models increasingly use A/B testing in dataset curation and synthesis(using GPT 4 level models) to optimize not just the benchmarks but things which could be benchmarked like academics.

Also "GPT-4 level" is a bit loaded. One way to think about it that I found helpful is to split how good a model is into "capability" and "knowledge/hallucination".

Many benchmarks test "capability" more than "knowledge". There are many use cases where the model gets all the necessary context in the prompt. There a model with good capability for the use case will do fine (e.g. as good as GPT-4).

That same model might hallucinate when you ask about the plot of a movie while a larger model like GPT-4 might be able to recall better what the movie is about.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#226

Post by larodi »

slics wrote: post #121

Warning- this is a lengthy comment. I deployed that model locally via Ollama and the reasoning model is not accurate. This is lengthy but I thought I share its reasoning- at some point it just stopped working - then it started responding in Chinese. Query input: “1+5 = 7, and that is the correct answer” Reasoning response: The equation "1 + 5 = 7" isn’t about regular arithmetic addition. Instead, it refers to the num…

bro, use pastebin for synthetic content pasted here. otherwise get downvoted as a rule of thumb.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#227

Post by int_19h »

elorant wrote: post #111
abraxas wrote: post #55

Earlier quoted context omitted.

Would a 40GB A6000 fully accommodate a 32B model? I assume an fp16 quantization is still necessary?

You don't need 16-bit quantization. The difference in accuracy from 8-bit in most models is less than 5%.

Even 4-bit is fine.

To be more precise, it's not that there's no decrease in quality, it's that with the RAM savings you can fit a much better model. E.g. with LLaMA, if you start with 70b and increasingly quantize, you'll still get considerably better performance at 3 bit than LLaMA 33b running at 8bit.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#229

Post by buyucu »

echelon wrote: post #17
simonw wrote: post #4

Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/

Pretty soon I won't be using any American models. It'll be a 100% Chinese open source stack. The foundation model companies are screwed. Only shovel makers (Nvidia, infra companies) and product companies are going to win.

OpenAI is basically a zombie company at this point. They could not make a profit even when they were the only player in town, it's now a very competitive landscape.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#230

Post by buyucu »

simonw wrote: post #6

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

I prefer 24b because it's the largest model I can run on a 16GB laptop :)
Post reply on HN