Qwen3 235B A22B GGUF bf16 is 470GB size lol
that's 3x h100?
Qwen3: Think deeper, act faster
31–40 of 412 posts
Re: Qwen3: Think deeper, act faster
#32These performance numbers look absolutely incredible. The MoE outperforms o1 with 3B active parameters? We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.
How do people typically do napkin math to figure out if their machine can “handle” a model?
The ultra-simplified napkin math is 1 GB (V)RAM per 1 billion parameters, at a 4-5 bit-per-weight quantization. This usually gives most of the performance of the full size model and leaves a little bit of room for context, although not necessarily the full supported size.
Re: Qwen3: Think deeper, act faster
#33These performance numbers look absolutely incredible. The MoE outperforms o1 with 3B active parameters? We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.
How do people typically do napkin math to figure out if their machine can “handle” a model?
Very rough (!) napkin math: for a q8 model (almost lossless) you have parameters = VRAM requirement. For q4 with some performance loss it's roughly half. Then you add a little bit for the context window and overhead. So a 32B model q4 should run comfortably on 20-24 GB.
Again, very rough numbers, there's calculators online.
Re: Qwen3: Think deeper, act faster
#34These performance numbers look absolutely incredible. The MoE outperforms o1 with 3B active parameters? We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.
How do people typically do napkin math to figure out if their machine can “handle” a model?
[deleted]
Re: Qwen3: Think deeper, act faster
#35They have got pretty good documentation too[1]. And Looks like we have day 1 support for all major inference stacks, plus so many size choices. Quants are also up because they have already worked with many community quant makers. Not even going into performance, need to test first. But what a stellar release just for attention to all these peripheral details alone. This should be the standard for major release, inste…
This cannot be stressed enough.
Re: Qwen3: Think deeper, act faster
#36Any news on some viable successor of LLMs that could take us to AGI? As I see they still can't solve some fundamental stuff to make it really work in any scenario (halucinations, reasoning, grounding in reality, updating long-term memory, etc.)
A mixture of many architectures. LLMs will probably play a part.
As for other possible technologies, I'm most excited about clone-structured causal graphs[1].
What's very special about them is that they are apparently a 1:1 algorithmic match to what happens in the hippocampus during learning[2], to my knowledge this is the first time an actual end-to-end algorithm has been replicated from the brain in fields other than vision.
[1] "Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps" https://www.nature.com/articles/s41467-021-22559-5
[2] "Learning produces an orthogonalized state machine in the hippocampus" https://www.nature.com/articles/s41586-024-08548-w
Re: Qwen3: Think deeper, act faster
#37With all the different open-weight models appearing, is there some way of figuring out what model would work with sensible speed (> X tok/s) on a standard desktop GPU ?
I.e. I have Quadro RTX 4000 with 8G vram and seeing all the models https://ollama.com/search here with all the different sizes, I am absolutely at loss which models with which sizes would be fast enough. I.e. there is no point of me downloading the latest biggest model as that will output 1 tok/min, but I also don't want to download the smallest model, if I can.
Any advice ?
Re: Qwen3: Think deeper, act faster
#38Something that interests me about the Qwen and DeepSeek models is that they have presumably been trained to fit the worldview enforced by the CCP, for things like avoiding talking about Tiananmen Square - but we've had access to a range of Qwen/DeepSeek models for well over a year at this point and to my knowledge this assumed bias hasn't actually resulted in any documented problems from people using the models.
Aside from https://huggingface.co/blog/leonardlin/chinese-llm-censorshi... I haven't seen a great deal of research into this.
Has this turned out to be less of an issue for practical applications than was initially expected? Are the models just not censored in the way that we might expect?
Re: Qwen3: Think deeper, act faster
#39Is this multimodal? They don’t mention it anywhere but if I go to QwenChat I can use images with it.
Re: Qwen3: Think deeper, act faster
#40>Hybrid Thinking Modes
This is what gpt-5 was supposed to have right? How is this implemented under the hood? Since non-thinking mode is just an empty chain-of-thought, why can't any reasoning model be used in a "non-thinking mode"?