Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

31–40 of 240 posts

Re: Qwen3-Next

#31

Post by irthomasthomas »

dev_hugepages wrote: post #23
irthomasthomas wrote: post #19

llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _…

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Naturally. That's how LLMs work. During training you measure the loss, the difference between the model output and the ground-truth and try to minimize it. We prize models for their ability to learn. Here we can see that the large model does a great job at learning to draw bob, while the small model performs poorly.

Re: Qwen3-Next

#33

Post by eurekin »

ginko wrote: post #29
dev_hugepages wrote: post #23

Earlier quoted context omitted.

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Conveniently removed the artist's signature though.

Certainly not defending LLMs here, don't mistake with that.

Humans do it too. I have given up on my country's non-local information sources, because I could recognize original sources that are being deliberately omitted. There's a satiric webpage that is basically a reddit scrape. Most of users don't notice and those who do, don't seem to care.

Re: Qwen3-Next

#34

Post by endymion-light »

irthomasthomas wrote: post #31
dev_hugepages wrote: post #23

Earlier quoted context omitted.

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Naturally. That's how LLMs work. During training you measure the loss, the difference between the model output and the ground-truth and try to minimize it. We prize models for their ability to learn. Here we can see that the large model does a great job at learning to draw bob, while the small model performs poorly.

I'd argue that actually, the smaller model is doing a better job at "learning" - in that it's including key characteristics within an ascii image while poor.

The larger model already has it in the training corpus so it's not particularly a good measure though. I'd much rather see the capabilities of a model in trying to represent in ascii something that it's unlikely to have in it's training.

Maybe a pelican riding a bike as ascii for both?

Re: Qwen3-Next

#35

Post by irthomasthomas »

ginko wrote: post #29
dev_hugepages wrote: post #23

Earlier quoted context omitted.

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Conveniently removed the artist's signature though.

Yes - they all do that. Actually, most attempts start well but unravel toward the end.

  llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  "An ASCII of spongebob"
  Here's an ASCII art of SpongeBob SquarePants:
  
  ```
      .--..--..--..--..--..--.
    .' \  (`._   (_)     _   \
  .'    |  '._)         (_)  |
  \ _.')\      .----..--.   /
  |(_.'  |    /    .-\-.  \
  \     0|    |   ( O| O) |
   |  _  |  .--.____.'._.-.
   /.' )  | (_.' .-'"`-. _.-._.-.--.-.
  / .''.  |  .' `-. .-'-. .-'"`-.`-._)
   .'.' |  |   |  |  |  |  |  |  |  |
  .'.'   |  |   |  |  |  |  |  |  |  |
  .'.'   |  |   |  |  |  |  |  |  |  |
  .'.'   |  |   |  |  |  |  |  |  |  |
  .'.'   |  |   |  |  |  |  |  |  |  |
  .'.'   |  |   |  |  |  |  |  |  |  |
  ```

Re: Qwen3-Next

#36

Post by vintermann »

NitpickLawyer wrote: post #11
vintermann wrote: post #8

Earlier quoted context omitted.

Could you give some practical examples? I don't know what Qwen's 36T-token training set is like, so I don't know what it's overfitting to...

Take math and coding for example: - in math, if they can solve a problem, or a class of problems, they'll solve it. If you use a "thinking" model + maj@x, you'll get strong results. But if you try for example to have the model consider a particular way or method of exploring a problem, it'll default to "solving" mode. It's near impossible to have it do something else with a math problem, other than solving it. Say "e…

Oof, that sounds frustrating. Yeah, I can relate to this failure mode, it's basically "did you mean (more likely query)" up to 11.

It does sound like an artifact of the dialog/thinking tuning though.

Re: Qwen3-Next

#37

Post by yorwba »

eurekin wrote: post #33
ginko wrote: post #29

Earlier quoted context omitted.

Conveniently removed the artist's signature though.

Certainly not defending LLMs here, don't mistake with that. Humans do it too. I have given up on my country's non-local information sources, because I could recognize original sources that are being deliberately omitted. There's a satiric webpage that is basically a reddit scrape. Most of users don't notice and those who do, don't seem to care.

Yes, the most likely reason the model omitted the signature is that humans reposted more copies of this image omitting the signature than ones that preserve it.

Re: Qwen3-Next

#39

Post by regularfry »

EnPissant wrote: post #27
yekanchi wrote: post #22

Earlier quoted context omitted.

i mean 4bit quantized. i can roughly calculate vram for dense models by model size. but i don't know how to do it for MOE models?

MoE models need just as much VRAM as dense models because every token may use a different set of experts. They just run faster.

This isn't quite right: it'll run with the full model loaded to RAM, swapping in the experts as it needs. It has turned out in the past that experts can be stable across more than one token so you're not swapping as much as you'd think. I don't know if that's been confirmed to still be true on recent MoEs, but I wouldn't be surprised.
Post reply on HN