Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-4B-Thinking-2507

huggingface.co

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Qwen3-4B-Thinking-2507

#51

Post by littlestymaar »

NitpickLawyer wrote: post #46
hnfong wrote: post #25

Earlier quoted context omitted.

Besides the LM Arena Leaderboard mentioned by a sibling comment, if go to the r/LocalLlama/ subreddit, you can very unscientifically get a rough sentiment of the performance of the models by reading the comments (and maybe even check the upvotes). I think the crowd's knee-jerk reaction is unreliable though, but that's what you asked for.

Not anymore tho. It used to be the place to vibe-check a model ~1 year ago, but lately it's filled with toxic my team vs. your team, memes about CEOs (wtf) and general poor takes on a lot of things. For a while it was china vs. world, but lately it's even more divided, with heavy camping on specific models. You can still get some signal, but you have to either ban a lot of accounts, or read new during different tzs s…

Yeah, some people just can't stop acting as if tech companies were sport teams, and it gets annoying fast.

Re: Qwen3-4B-Thinking-2507

#52

Post by Gracana »

magnat wrote: post #45
nisten wrote: post #36

If you want to have an opinion on it, just install lmstudio and run the q8_0 version of it i.e. here https://huggingface.co/bartowski/Qwen_Qwen3-4B-Instruct-2507... . you can even run it on a 4gb raspberry pi Qwen_Qwen3-4B-Instruct-2507-Q4_K_L.gguf https://lmstudio.ai/ Keep in mind if you run it at the full 262144 tokens of context youll need ~65gb of ram. Anyway if you're on mac you can search for "qwen3 4b 2507 mlx…

> if you run it at the full 262144 tokens of context youll need ~65gb of ram What is the relationship between context size and RAM required? Isn't the size of RAM related only to number of parameters and quantization?

The context cache (or KV cache) is where intermediate results are stored. One for each output token. Its size depends on the model architecture and dimensions.

KV cache size = 2 * batch_size * context_len * num_key_value_heads * head_dim * num_layers * element_size. The "2" is for the two parts, key and value. Element size is the precision in bytes. This model uses grouped query attention, which reduces num_key_value_heads compared to a multi head attention (MHA) model.

With batch size 1 (for low-latency single-user inference), 32k context (recommended in the model card), fp16 precision:

2 * 1 * 32768 * 8 * 128 * 36 * 2 = 4.5GiB.

I think, anyway. It's hard to keep up with this stuff. :)

Re: Qwen3-4B-Thinking-2507

#53

Post by hnuser123456 »

magnat wrote: post #45
nisten wrote: post #36

If you want to have an opinion on it, just install lmstudio and run the q8_0 version of it i.e. here https://huggingface.co/bartowski/Qwen_Qwen3-4B-Instruct-2507... . you can even run it on a 4gb raspberry pi Qwen_Qwen3-4B-Instruct-2507-Q4_K_L.gguf https://lmstudio.ai/ Keep in mind if you run it at the full 262144 tokens of context youll need ~65gb of ram. Anyway if you're on mac you can search for "qwen3 4b 2507 mlx…

> if you run it at the full 262144 tokens of context youll need ~65gb of ram What is the relationship between context size and RAM required? Isn't the size of RAM related only to number of parameters and quantization?

A 24GB GPU can run a ~30b parameter model at 4bit quantization at about 8k-12k context length before every GB of VRAM is occupied.

Re: Qwen3-4B-Thinking-2507

#54

Post by [deleted] »

byefruit wrote: post #9
cowpig wrote: post #7

Earlier quoted context omitted.

Compare these rankings to actual usage: https://openrouter.ai/rankings Claude is not cheap, why is it far and away the most popular if it's not top 10 in performance? Qwen3 235b ranks highest on these benchmarks among open models, but I have never met someone who prefers its output over Deepseek R1. It's extremely wordy and often gets caught in thought loops. My interpretation is that the models at the top of Artific…

The openrouter rankings can be biased. For example, Google's inexplicable design decisions around libraries and APIs means it's often worth the 5% premium to just use OpenRouter to access their models. In other cases it's about which models particular agents default to. Sonnet 4 is extremely good for tool-usage agentic setups though - something I have found other models struggle to do over a long-context.

[deleted]

Re: Qwen3-4B-Thinking-2507

#55

Post by belter »

esafak wrote: post #42
nisten wrote: post #36

If you want to have an opinion on it, just install lmstudio and run the q8_0 version of it i.e. here https://huggingface.co/bartowski/Qwen_Qwen3-4B-Instruct-2507... . you can even run it on a 4gb raspberry pi Qwen_Qwen3-4B-Instruct-2507-Q4_K_L.gguf https://lmstudio.ai/ Keep in mind if you run it at the full 262144 tokens of context youll need ~65gb of ram. Anyway if you're on mac you can search for "qwen3 4b 2507 mlx…

Thank you. To spare Mac readers time: mlx 4bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-... mlx 5bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-... mlx 6bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-... mlx 8bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-... edit: corrected the 4b link

This comment saved 3 tons of CO2

Re: Qwen3-4B-Thinking-2507

#56

Post by 0x457 »

magnat wrote: post #45
nisten wrote: post #36

If you want to have an opinion on it, just install lmstudio and run the q8_0 version of it i.e. here https://huggingface.co/bartowski/Qwen_Qwen3-4B-Instruct-2507... . you can even run it on a 4gb raspberry pi Qwen_Qwen3-4B-Instruct-2507-Q4_K_L.gguf https://lmstudio.ai/ Keep in mind if you run it at the full 262144 tokens of context youll need ~65gb of ram. Anyway if you're on mac you can search for "qwen3 4b 2507 mlx…

> if you run it at the full 262144 tokens of context youll need ~65gb of ram What is the relationship between context size and RAM required? Isn't the size of RAM related only to number of parameters and quantization?

I mean...where do you think context is stored?

Re: Qwen3-4B-Thinking-2507

#57

Post by parineum »

NitpickLawyer wrote: post #46
hnfong wrote: post #25

Earlier quoted context omitted.

Besides the LM Arena Leaderboard mentioned by a sibling comment, if go to the r/LocalLlama/ subreddit, you can very unscientifically get a rough sentiment of the performance of the models by reading the comments (and maybe even check the upvotes). I think the crowd's knee-jerk reaction is unreliable though, but that's what you asked for.

Not anymore tho. It used to be the place to vibe-check a model ~1 year ago, but lately it's filled with toxic my team vs. your team, memes about CEOs (wtf) and general poor takes on a lot of things. For a while it was china vs. world, but lately it's even more divided, with heavy camping on specific models. You can still get some signal, but you have to either ban a lot of accounts, or read new during different tzs s…

I don't really go there much anymore but, when I was, there seemed to be an innordinate amount of Chinese nationalism from young accounts speaking odd English.

Re: Qwen3-4B-Thinking-2507

#58

Post by parineum »

evilduck wrote: post #37
Workaccount2 wrote: post #29

Earlier quoted context omitted.

China has a business model where you can lose money and it doesn't matter. The state's modus operandi is just fund things until the leader changes his mind about it. This is why the Chinese labs are so open, they don't ever need to make a profit, they just need to make good AI.

Sure. The US isn't morally any better on this though. VC backed companies and the "get users, figure out profitability later" strategy that have made outside countries unable to compete have been enabled by the US government tweaking interest rates and the supply value of the dollar. And the US backs all sorts of unprofitable things through grants, contracts, and bail outs. The US government just hasn't yet found a r…

> US government tweaking interest rates

They actually don't do that and, if you're up to speed on current events, Trump is pretty upset about it.

Re: Qwen3-4B-Thinking-2507

#60

Post by wkat4242 »

Gracana wrote: post #52
magnat wrote: post #45

Earlier quoted context omitted.

> if you run it at the full 262144 tokens of context youll need ~65gb of ram What is the relationship between context size and RAM required? Isn't the size of RAM related only to number of parameters and quantization?

The context cache (or KV cache) is where intermediate results are stored. One for each output token. Its size depends on the model architecture and dimensions. KV cache size = 2 * batch_size * context_len * num_key_value_heads * head_dim * num_layers * element_size. The "2" is for the two parts, key and value. Element size is the precision in bytes. This model uses grouped query attention, which reduces num_key_value…

Yes but you can quantise the KV cache too just like you can the weights.
Post reply on HN