Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen2.5-VL-32B: Smarter and Lighter

qwenlm.github.io

Post reply on HN
Print view View original

241–250 of 303 posts

Re: Qwen2.5-VL-32B: Smarter and Lighter

#241

Post by rfoo »

pzo wrote: post #212
energyrace wrote: post #168

Earlier quoted context omitted.

What's your take on why certain countries will have it cheaper and subsidies being at the forefront? An energy driven race to the bottom, is perhaps what you mean? I would suppose I have been seeing that China is ahead on their Renewables plan compared to the rest of the world, and they still have the lead on coal energy, so they'd likely be the winners on that front. But did you actually mean something else?

The problem with china is, they will have to figure out latency. Right now DeepSeek models hosted in china are having very high latency. It could because of DDoS and not strong enough infrastructure but probably also because of Great Firewall, runtime censoring prompt and servers physical location (big ping to US and EU countries).

> Right now DeepSeek models hosted in china are having very high latency.

If you are talking about DeepSeek's own hosted API service. It's because they deliberately decided to run the service in heavily overloaded conditions and have very aggressive batching policy to extract more out of their (limited) H800s.

Yes, for some reason (the reason I heard is "our boss don't want to run such a business" which sounds absurd but /shrug) they refuse to scale up serving their own models.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#242

Post by piokoch »

echelon wrote: post #17
simonw wrote: post #4

Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/

Pretty soon I won't be using any American models. It'll be a 100% Chinese open source stack. The foundation model companies are screwed. Only shovel makers (Nvidia, infra companies) and product companies are going to win.

"The foundation model companies are screwed." Not really, they can either make API access expensive or resign from exposing APIs and offer their custom products. Open Source models are great, but you need powerful hardware to run them, surely it will not be a smartphone, at least in the nearest future.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#243

Post by Anaphylaxis »

slics wrote: post #121

Warning- this is a lengthy comment. I deployed that model locally via Ollama and the reasoning model is not accurate. This is lengthy but I thought I share its reasoning- at some point it just stopped working - then it started responding in Chinese. Query input: “1+5 = 7, and that is the correct answer” Reasoning response: The equation "1 + 5 = 7" isn’t about regular arithmetic addition. Instead, it refers to the num…

Almost guaranteed this is user error. Ollama has a (tiny) default of 2048 context, so about then is probably when you noticed the results sharply decline in quality. Try 16384

Re: Qwen2.5-VL-32B: Smarter and Lighter

#244

Post by regularfry »

faizshah wrote: post #35
simonw wrote: post #6

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

I just started self hosting as well on my local machine, been using https://lmstudio.ai/ Locally for now. I think the 32b models are actually good enough that I might stop paying for ChatGPT plus and Claude. I get around 20 tok/second on my m3 and I can get 100 tok/second on smaller models or quantized. 80-100 tok/second is the best for interactive usage if you go above that you basically can’t read as fast as it gen…

Qwq:32b + qwen2.5-coder:32b is a nice combination for aider, running locally on a 4090. It has to swap models between architect and edit steps so it's not especially fast, but it's capable enough to be useful. qwen2.5-coder does screw up the edit format sometimes though, which is a pain.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#245

Post by regularfry »

Auracle wrote: post #164
TrueDuality wrote: post #124

Earlier quoted context omitted.

AMD's limitation is more of a software problem than a hardware problem at this point.

But it’s still surprising they haven’t. People would be motivated as hell if they launched GPUs with twice the amount of VRAM. It’s not as simple as just soldering some more in but still.

Funnily enough you can buy GPUs where someone has done exactly that: solder extra VRAM into a stock model.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#246

Post by regularfry »

p_l wrote: post #81
wetwater wrote: post #38

Earlier quoted context omitted.

Are there any good sources that I can read up on estimiating what would be hardware specs required for 7B, 13B, 32B .. etc size If I need to run them locally? I am grad student on budget but I want to host one locally and trying to build a PC that could run one of these models.

Generally, unquantized - double the number and that's the amount of VRAM in GB you need + some extra, because most models use fp16 weights so it's 2 bytes per parameter -> 32B parameters = 64GB typical quantization to 4bit will cut 32B model into 16GB of weights plus some of the runtime data, which makes it possibly usable (if slow) on 16GB GPU. You can sometimes viably use smaller quantizations, which will reduce me…

You always want a bit of headroom for context. It's a problem I keep bumping into with 32B models on a 24GB card: the decent quants fit, but the context you have available on the card isn't quite as much as I'd like.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#247

Post by regularfry »

osti wrote: post #197
simonw wrote: post #6

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

Are 5090's able to run 32B models?

The 4090 can run 32B models in Q4_K_M, so yes, on that measure. Not unquantised though, nothing bigger than Q8 would fit. On a 32GB card you'll have more choices to trade off quantisation against context.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#250

Post by submeta »

voidUpdate wrote: post #248
submeta wrote: post #211

Heavily censors certain political topics (in the Middle East). Grok is doing none of this. How is this possible, coming from China? What stakes got China in the ME conflict?

Does Grok still deny that trans women are women?

That’s a red herring. The original point was about censorship on Middle East topics.
Post reply on HN