121–130 of 178 posts
Earlier quoted context omitted.
I am using the Q6_K_L quant and it's running at about 40G of vram with the KV cache. Device 1 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||20.170Gi/23.988Gi] Device 2 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||19.945Gi/23.988Gi]
What's the context length?
Chinese strategy is open-source software part and earn on robotics part. And, They are already ahead of everyone in that game. These things are pretty interesting as they are developing. What US will do to retain its power? BTW I am Indian and we are not even in the race as country. :(
This is ridiculous. 32B and beating deepseek and o1. And yet I'm trying it out and, yeah, it seems pretty intelligent... Remember when models this size could just about maintain a conversation?
It says "wait" (as in "wait, no, I should do X") so much while reasoning it's almost comical. I also ran into the "catastrophic forgetting" issue that others have reported - it sometimes loses the plot after producing a lot of reasoning tokens. Overall though quite impressive if you're not in a hurry.
Is the model using budget forcing?
why not just decrease probability? if model really wants to finish maybe or could over power it in cases were it's really simple question. and definitely would allow model to express next thought more freely
https://gist.github.com/davidmezzetti/049d3078e638aa8497b7cd...
Last time I tried QwQ or QvQ (a couple of days ago), their CoT was so long that it almost seemed endless, like it was stuck in a loop. I hope this doesn't have the same issue.
Last time I tried QwQ or QvQ (a couple of days ago), their CoT was so long that it almost seemed endless, like it was stuck in a loop. I hope this doesn't have the same issue.
It's a method used to control thinking token generation showcased in this paper: https://arxiv.org/abs/2501.19393
Earlier quoted context omitted.
The quantized model fits in about 20 GB, so 32 would probably be sufficient unless you want to use the full context length (long inputs and/or lots of reasoning). 48 should be plenty.
I‘ve tried the very early Q4 mlx release on an M1 Max 32GB (LM Studio @ default settings), and have run into severe issues. For the coding tasks I gave it, it froze before it was done with reasoning. I guess I should limit context size. I do love what I‘m seeing though, the output reads very similar to R1, and I mostly agree with its conclusions. The Q8 version has to be way better even.
Earlier quoted context omitted.
I‘ve tried the very early Q4 mlx release on an M1 Max 32GB (LM Studio @ default settings), and have run into severe issues. For the coding tasks I gave it, it froze before it was done with reasoning. I guess I should limit context size. I do love what I‘m seeing though, the output reads very similar to R1, and I mostly agree with its conclusions. The Q8 version has to be way better even.
Does the Q8 fit within your 32GB (also using an M1 32GB)