Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3: Think deeper, act faster

qwenlm.github.io

Post reply on HN
Print view View original

281–290 of 412 posts

Re: Qwen3: Think deeper, act faster

#281

Post by BoredPositron »

mrkeen wrote: post #253
stavros wrote: post #98

I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer correctly. If I ask them to explain their answer, they do get it eventually, but none get it right the first time. Qwen3 with max thinking got it even more wrong than the rest, for what it's worth.

As they say, we shouldn't judge AI by the current state-of-the-art, but by how far and fast it's progressing. I can't wait to see future models get it even more wrong than that.

Haven’t seen much progress in base models since gpt4. Deep thinking and whatever else came in the last year are just bandaids hiding the shortcomings of said models and were achievable before with the right tooling. The tooling got better the models themselves are just marginally better.

Re: Qwen3: Think deeper, act faster

#282

Post by PhilippGille »

sega_sai wrote: post #37

With all the different open-weight models appearing, is there some way of figuring out what model would work with sensible speed (> X tok/s) on a standard desktop GPU ? I.e. I have Quadro RTX 4000 with 8G vram and seeing all the models https://ollama.com/search here with all the different sizes, I am absolutely at loss which models with which sizes would be fast enough. I.e. there is no point of me downloading the la…

Mozilla started LocalScore for exactly what you're looking for: https://www.localscore.ai/

Re: Qwen3: Think deeper, act faster

#283

Post by estsauver »

sega_sai wrote: post #37

With all the different open-weight models appearing, is there some way of figuring out what model would work with sensible speed (> X tok/s) on a standard desktop GPU ? I.e. I have Quadro RTX 4000 with 8G vram and seeing all the models https://ollama.com/search here with all the different sizes, I am absolutely at loss which models with which sizes would be fast enough. I.e. there is no point of me downloading the la…

I don't think this is all that well documented anywhere. I've had this problem too and I don't think anyone has tried to record something like a decent benchmark of token inference/speed for a few different models. I'm going to start doing it while playing around with settings a bit. Here's some results on my (big!) M4 Mac Pro with Gemma 3, I'm still downloading Qwen3 but will update when it lands.

https://gist.github.com/estsauver/a70c929398479f3166f3d69bce...

Here's a video of the second config run I ran so you can see both all of the parameters as I have them configured and a qualitative experience.

https://screen.studio/share/4VUt6r1c

Re: Qwen3: Think deeper, act faster

#284

Post by tamat »

furyofantares wrote: post #145
stavros wrote: post #98

I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer correctly. If I ask them to explain their answer, they do get it eventually, but none get it right the first time. Qwen3 with max thinking got it even more wrong than the rest, for what it's worth.

I similarly have a small, simple spatial reasoning problem that only reasoning models get right, and not all of them, and which Qwen3 on max reasoning still gets wrong. > I put a coin in a cup and slam it upside-down on a glass table. I can't see the coin because the cup is over it. I slide a mirror under the table and see heads. What will I see if I take the cup (and the mirror) away?

I always feel that if you share a problem here where LLMs fail, it will end up in their training set and it wont fail to that problem anymore, which means the future models will have the same errors but you have lost your ability to detect them.

Re: Qwen3: Think deeper, act faster

#285

Post by horhay »

nopinsight wrote: post #222
stavros wrote: post #98

I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer correctly. If I ask them to explain their answer, they do get it eventually, but none get it right the first time. Qwen3 with max thinking got it even more wrong than the rest, for what it's worth.

Current models are quite far away from human-level physical reasoning (paper below). An upcoming version of models trained on world simulation will probably do much better. PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models https://phybench-official.github.io/phybench-demo/

This is more about a physics math aptitude test. You can already see that the best model in math is saturating it halfway. It might not indicate its usefulness in actual physical reasoning, or at the very least, it seems like a bit of a stretch.

Re: Qwen3: Think deeper, act faster

#286

Post by hobofan »

manmal wrote: post #276
simonw wrote: post #207

As is now traditional for new LLM releases, I used Qwen 3 (32B, run via Ollama on a Mac) to summarize this Hacker News conversation about itself - run at the point when it hit 112 comments. The results were kind of fascinating, because it appeared to confuse my system prompt telling it to summarize the conversation with the various questions asked in the post itself, which it tried to answer. I don't think it did a g…

One person on Reddit claimed the first unsloth release was buggy - if you used that, maybe you can retry with the fixed version?

I think this was only regarding the chat template that was provided in the metadata (this was also broken in the official release). However, I doubt that this would impact this test, as most inference frameworks will just error if provided with a broken template.

Re: Qwen3: Think deeper, act faster

#287

Post by horhay »

bergheim wrote: post #151
echelon wrote: post #93

Earlier quoted context omitted.

Alibaba, I have a huge favor to ask if you're listening. You guys very obviously care about the community. We need an answer to gpt-image-1. Can you please pair Qwen with Wan? That would literally change the art world forever. gpt-image-1 is an almost wholesale replacement of ComfyUI and SD/Flux ControlNets. I can't underscore how big of a deal it is. As such, OpenAI has leapt ahead and threatens to start capturing m…

> That would literally change the art world forever. In what world? Some small percentage up or who knows, and _that_ revolutionized art? Not a few years ago, but now, this. Wow.

It's pretty much expected that everything is "world shaking" in the modern day tech world. Now whether it's true or not is a different thing everytime. I'm fairly certain even the 4o image gen model has shown weaknesses that other approaches didn't, but you know, newer means absolutely better and will change the world.

Re: Qwen3: Think deeper, act faster

#288

Post by ggregoryarms »

jim180 wrote: post #274
kaoD wrote: post #256

Earlier quoted context omitted.

Personally (anecdata) I haven't experienced any practical progress in my day-to-day tasks for a long time, no matter how good they became at gaming the benchmarks. They keep being impressive at what they're good at (aggregating sources to solve a very well known problem) and terrible at what they're bad at (actually thinking through novel problems or old problems with few sources). E.g. all ChatGPT, Claude and Gemini…

Absolutely. All models ar terrible with Objective-C and Swift, compared to let's say JS/HTML/Python. However, I've realized that Claude Code is extremely useful for generating somewhat simple landing pages for some of my projects. It spits out static html+js which is easy to host, with somewhat good looking design. The code isn't the best and to some extent isn't maintainable by a human at all, but it gets the job do…

Building a basic static html landing page is ridiculously easy though. What js is even needed? If it's just an html file and maybe a stylesheet of course it's easy to host. You can apply 20 lines of css and have a decent looking page.

These aren't hard problems.

Re: Qwen3: Think deeper, act faster

#289

Post by Imustaskforhelp »

echelon wrote: post #93
Jayakumark wrote: post #53

Earlier quoted context omitted.

Second this , they patched all major llm frameworks like llama.cpp, transformers , vllm, sglang, ollama etc weeks before for qwen3 support and released model weights everywhere around same time. Like a global movie release. Cannot undermine mine this level of detail and effort.

Alibaba, I have a huge favor to ask if you're listening. You guys very obviously care about the community. We need an answer to gpt-image-1. Can you please pair Qwen with Wan? That would literally change the art world forever. gpt-image-1 is an almost wholesale replacement of ComfyUI and SD/Flux ControlNets. I can't underscore how big of a deal it is. As such, OpenAI has leapt ahead and threatens to start capturing m…

I don't know, the AI image quality has gotten good but it's still slop. We are forgetting what makes art, well art.

I am not even an artist but yeah I see people using AI for photos and they were so horrendous pre chatgpt-imagen that I had literally told one person if you are going to use AI images, might as well use chatgpt for it.

Also though I would also like to get something like chatgpt-image generating qualities from an open source model. I think what we are really looking for is cheap free labour of alibaba team.

We are wanting for them / anyone to create open source tool so that anyone can then use it, thus reducing the monopoly of openai but that is not what most people are wishing for, they are wishing for this to lead to reduction of price so that they can use it either on their own hardware for very few cost or some providers on openrouter and its alikes for cheap image generation with good quality.

Earlier people used to pay artists, then people started using stock photos, then Ai image gen came, and now we have gotten AI image pretty much good with chatgpt and now people don't even want to pay chatgpt that much money, they want to use it for literal cents.

Not sure how long this trend will continue, when deepseek r1 launched, I remember people being happy that it was open source but 99% people couldn't self host it like I can't because of its needs and we were still using API but just because it was open source, it reduced the price way too much forcing others to reduce it as well, really making a cultural pricing shift in AI.

We are in this really weird spot as humans. We want to earn a lot of money yet we don't want to pay anybody money/ want free labour from open source which is just disincentivizing open source because now people like to think its free labour and they might be right.

Re: Qwen3: Think deeper, act faster

#290

Post by ggregoryarms »

animal531 wrote: post #280
stavros wrote: post #98

I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer correctly. If I ask them to explain their answer, they do get it eventually, but none get it right the first time. Qwen3 with max thinking got it even more wrong than the rest, for what it's worth.

They all are using these tests to determine their worth, but to be honest they don't convert well to real world tests. For example I tried Deepseek for code daily over a period of about two months (vs having used ChatGPT before), and its output was terrible. It would produce code with bugs, break existing code when making additions, totally fail at understanding what you're asking etc.

Exactly. If I'm going to be solving bugs, I'd rather they be my own.
Post reply on HN