[deleted]
QwQ: Alibaba's O1-like reasoning LLM
341–350 of 435 posts
Re: QwQ: Alibaba's O1-like reasoning LLM
#342Earlier quoted context omitted.
Runs nicely on my AMD 7900XTX too.
How are AMD cards performing? I heard it was still very hit and miss in regards to support. Have you also tried things like F5-TTS?
7900XT here, previously 6900XT. Support for 6000 series and beyond is great on Linux. You have to use an override envar (Ollama has info on this in their readme). ComfyUI has instructions for anything based on torch.
TensorFlow is unusable.
Re: QwQ: Alibaba's O1-like reasoning LLM
#343Earlier quoted context omitted.
Runs nicely on my AMD 7900XTX too.
Care to say how many tokens per second you're getting?
For the "How many r's in the word strawberry"
total duration: 15.278476756s
load duration: 14.982999ms
prompt eval count: 47 token(s)
prompt eval duration: 5ms
prompt eval rate: 9400.00 tokens/s
eval count: 377 token(s)
eval duration: 15.257s
eval rate: 24.71 tokens/sRe: QwQ: Alibaba's O1-like reasoning LLM
#344I’m tried it and it keeps refusing to answer coding questions. It just says I cannot answer that.
Re: QwQ: Alibaba's O1-like reasoning LLM
#345It still fails at very simple stuff.
E.g. "I put an ordinary rock into a glass of water. I then turn the glass of water upside down, do a little dance, and then turn the glass right side up again. Where is the rock now?"
100+ lines later...
"The rock is at the bottom of the glass, submerged in the water."
Models from a year ago get this right sometimes
Re: QwQ: Alibaba's O1-like reasoning LLM
#34632B is a good choice of size, as it allows running on a 24GB consumer card at ~4 bpw (RTX 3090/4090) while using most of the VRAM. Unlike llama 3.1, which had 8b, 70B (much too big to fit), and 405B.
what do you mean? I can easily run 70b on my macbook. Fits easily.
Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or Apple's unified memory will never perform as well as dedicated GPU memory.
Re: QwQ: Alibaba's O1-like reasoning LLM
#347We are lucky that Alibaba, Meta and Mistral sees some strategic value in public releases. If we it was just one of them, it would be a fragile situation for downstream startups. And they’re even situated in three different countries.
> We are lucky that Alibaba, Meta and Mistral sees some strategic value in public releases. Now if we only can get Meta to understand what "Open Source" means so the word doesn't lose all meaning in the future.
Words mean whatever's convenient to the bottom line, which is why the OSI (a consortium of Amazon, Google, Microsoft etc) still doesn't recognize the SSPL, as it would be particularly inconvenient for clouds.
Re: QwQ: Alibaba's O1-like reasoning LLM
#348Earlier quoted context omitted.
what do you mean? I can easily run 70b on my macbook. Fits easily.
Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or Apple's unified memory will never perform as well as dedicated GPU memory.
> shared memory architectures like Intel's integrated GPUs or Apple's unified memory can't compare to dedicated GPU memory.
It depends. For the initial prompt+context processing, iGPUs (and NPUs) are still a win because you're compute bound at that stage. For token generation, CPU is typically faster than iGPU (or even plausibly NPU) for local quantized models because it can make better use of all available memory bandwidth, with no wasteful padding reducing your usable throughput.
Re: QwQ: Alibaba's O1-like reasoning LLM
#349Earlier quoted context omitted.
Most of the papers in machine learning are coming from China. The vast majority. Most of the open source models on GitHub, too. Hailuo, Kling, Vidu, and Hunyuan (posted on Banodoko) blow Sora and Runway out of the water. China is dominating at this field. And if they begin releasing weights as open source, that'll mean foundation model companies can only bank on the thin facade of product. That's a really good strate…
I don't know enough to comment but can someone downvoting this please explain why they are doing so? Is it untrue, an exaggeration, something else or maybe simply an observation they don't like to see.
People really don't like to see that China has surpassed the USA in technology. It was the USA's main advantage - that and firepower.
Re: QwQ: Alibaba's O1-like reasoning LLM
#350Earlier quoted context omitted.
When you say this, I don't think any Chinese people actually believe you.
Not sure if the irony is intended here. The entire point is that the Chinese people aren't a monolith, hence CCP != The Chinese people. This will also hold for whether they believe us - in that too, Chinese people won't be a monolith. Plenty of those who aren't the biggest fans of the CCP will, as they understand where we're coming from better than anyone.
>This will also hold for whether they believe us - in that too, Chinese people won't be a monolith. Plenty of those who aren't the biggest fans of the CCP will, as they understand where we're coming from better than anyone.
Same can be said about people in US or most countries. It's just the big countries gets pumped up by the media and entertainment industry, so people tend to relate to them in a monolithic way.