Qwen2.5-VL-32B: Smarter and Lighter
qwenlm.github.io
Qwen2.5-VL-32B: Smarter and Lighter
1–10 of 303 posts
Re: Qwen2.5-VL-32B: Smarter and Lighter
#2Wish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB?
I guess the -7B might run on my 16GB AMD card?
Re: Qwen2.5-VL-32B: Smarter and Lighter
#3Wish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB? I guess the -7B might run on my 16GB AMD card?
You can run 4-bit quantized version at a small (though nonzero) cost to output quality, so you would only need 16GB for that.
Also it's entirely possible to run a model that doesn't fit in available GPU memory, it will just be slower.
Re: Qwen2.5-VL-32B: Smarter and Lighter
#4Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/
Re: Qwen2.5-VL-32B: Smarter and Lighter
#5Wish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB? I guess the -7B might run on my 16GB AMD card?
I wish they would start producing graphs with quantized version performances as well. What matters is RAM/bandwidth vs performance, not number of parameters.
Re: Qwen2.5-VL-32B: Smarter and Lighter
#632B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).
Re: Qwen2.5-VL-32B: Smarter and Lighter
#732B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).
32B don't fully fit 16GB of VRAM. Still fine for higher quality answers, worth the extra wait in some cases.
Re: Qwen2.5-VL-32B: Smarter and Lighter
#8Wish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB? I guess the -7B might run on my 16GB AMD card?
deepseek-r1:14b/mistral-small:24b/qwen2.5-coder:14b fit 16GB VRAM with fast generation.
32b versions bleed into RAM and take a serious performance hit but still usable.
Re: Qwen2.5-VL-32B: Smarter and Lighter
#9Big day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/
it seems that this free version "may use your prompts and completions to train new models"
https://openrouter.ai/deepseek/deepseek-chat-v3-0324:free
do you think this needs attention?
Re: Qwen2.5-VL-32B: Smarter and Lighter
#1032B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).
Or quantized on a 4090!