Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen2.5-VL-32B: Smarter and Lighter

qwenlm.github.io

Post reply on HN
Print view View original

81–90 of 303 posts

Re: Qwen2.5-VL-32B: Smarter and Lighter

#81

Post by p_l »

wetwater wrote: post #38
faizshah wrote: post #35

Earlier quoted context omitted.

I just started self hosting as well on my local machine, been using https://lmstudio.ai/ Locally for now. I think the 32b models are actually good enough that I might stop paying for ChatGPT plus and Claude. I get around 20 tok/second on my m3 and I can get 100 tok/second on smaller models or quantized. 80-100 tok/second is the best for interactive usage if you go above that you basically can’t read as fast as it gen…

Are there any good sources that I can read up on estimiating what would be hardware specs required for 7B, 13B, 32B .. etc size If I need to run them locally? I am grad student on budget but I want to host one locally and trying to build a PC that could run one of these models.

Generally, unquantized - double the number and that's the amount of VRAM in GB you need + some extra, because most models use fp16 weights so it's 2 bytes per parameter -> 32B parameters = 64GB

typical quantization to 4bit will cut 32B model into 16GB of weights plus some of the runtime data, which makes it possibly usable (if slow) on 16GB GPU. You can sometimes viably use smaller quantizations, which will reduce memory use even more.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#82

Post by postalrat »

manmal wrote: post #72
abraxas wrote: post #55

Earlier quoted context omitted.

Would a 40GB A6000 fully accommodate a 32B model? I assume an fp16 quantization is still necessary?

At FP16 you‘d need 64GB just for the weights, and it‘d be 2x as slow as a Q8 version, likely with little improvement. You‘ll also need space for attention and context etc, so 80-100GB (or even more) VRAM would be better. Many people „just“ use 4x consumer GPUs like the 3090 (24GB each) which scales well. They’d probably buy a mining rig, EPYC CPU, Mainboard with sufficient PCIe lanes, PCIe risers, 1600W PSU (might ne…

I haven't found a good case/risers/etc I really like. Most the miner stuff wasn't made for PCIe 16x.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#83

Post by 101008 »

barbarr wrote: post #78
101008 wrote: post #65

Silly question: how can OpenAI, Claude and all, have a valuation so large considering all the open source models? Not saying they will disappear or be tiny (closed models), but why so so so valuable?

It's user base and brand. Just like with Pepsi and Coca Cola. There's a reason OpenAI ran a Super Bowl ad.

Yeah but cheaper alternatives (and open source and local ones) it would be super easy for most of the customers to migrate to a different provider. I am not saying they don't provide any value, but it's like paid software vs open source alternative. Open source alternative ends up imposing, especially among tech people.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#84

Post by th0ma5 »

simonw wrote: post #20
YetAnotherNick wrote: post #13

Earlier quoted context omitted.

I don't think these models are GPT-4 level. Yes they seem to be on benchmarks, but it has been known that models increasingly use A/B testing in dataset curation and synthesis(using GPT 4 level models) to optimize not just the benchmarks but things which could be benchmarked like academics.

I'm not talking about GPT-4o here - every benchmark I've seen has had the new models from the past ~12 months out-perform the March 2023 GPT-4 model. To pick just the most popular one, https://lmarena.ai/?leaderboard= has GPT-4-0314 ranked 83rd now.

How have you been able to tie benchmark results to better results?

Re: Qwen2.5-VL-32B: Smarter and Lighter

#85

Post by th0ma5 »

simonw wrote: post #4

Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/

Why do you keep promoting your blog on every LLM post?

Re: Qwen2.5-VL-32B: Smarter and Lighter

#86

Post by ggregoire »

We were using Llama vision 3.2 a few months back and were very frustrated with it (both in term of speed and results quality). Some day we were looking for alternatives on Hugging Face and eventually stumbled upon Qwen. The difference in accuracy and speed absolutely blew our mind. We ask it to find something in an image and we get a response in like half a second with a 4090 and it's most of the time correct. What's even more mind blowing is that when we ask it to extract any entity name from the image, and the entity name is truncated, it gives us the complete name without even having to ask for it (e.g. "Coca-C" is barely visible in the background, it will return "Coca-Cola" on its own). And it does it with entities not as well known as Coca-Cola, and with entities only known in some very specific regions too. Haven't looked back to Llama or any other vision models since we tried Qwen.

Re: Qwen2.5-VL-32B: Smarter and Lighter

#87

Post by seydor »

101008 wrote: post #65

Silly question: how can OpenAI, Claude and all, have a valuation so large considering all the open source models? Not saying they will disappear or be tiny (closed models), but why so so so valuable?

People cannot normally invest in their competitors.

It's not unlikely that chinese products may be banned / tarriff'd

Re: Qwen2.5-VL-32B: Smarter and Lighter

#88

Post by anon373839 »

OsrsNeedsf2P wrote: post #80
gatienboquet wrote: post #11

So today is Qwen. Tomorrow a new SOTA model from Google apparently, R2 next week. We haven't hit the wall yet.

> We haven't hit the wall yet. The models are iterative improvements, but I haven't seen night and day differences since GPT3 and 3.5

Yeah. Scaling up pretraining and huge models appears to be done. But I think we're still advancing the frontier in the other direction -- i.e., how much capability and knowledge can we cram into smaller and smaller models?

Re: Qwen2.5-VL-32B: Smarter and Lighter

#89

Post by azinman2 »

zamadatix wrote: post #56
jauntywundrkind wrote: post #2

Wish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB? I guess the -7B might run on my 16GB AMD card?

https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... That will help you quickly calculate the model VRAM usage as well as the VRAM usage of the context length you want to use. You can put "Qwen/Qwen2.5-VL-32B-Instruct" in the "Model (unquantized)" field. Funnily enough the calculator lacks the option to see without quantizing the model, usually because nobody worried about VRAM bothers running >8 bit quant…

Except when it comes to deepseek

Re: Qwen2.5-VL-32B: Smarter and Lighter

#90

Post by theturtletalks »

hmottestad wrote: post #59
theturtletalks wrote: post #50

Earlier quoted context omitted.

Wouldn’t they just run R1 locally and not have any censorship at all? The model isn’t censored at its core, it’s censored through the system prompt. Perplexity and Huggingface have their own versions of R1 that is not censored.

I tried R1 through Kagi and it’s similarly censored. Even the distill of llama running on Groq is censored.

Kagi may be using the official DeepSeek API and not hosting the model itself. There is work being done to make it completely uncensored:

https://github.com/huggingface/open-r1

https://ollama.com/huihui_ai/deepseek-r1-abliterated

I was mistaken though, it is more than just a system prompt causing the censorship.

Post reply on HN