Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

github.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#31

Post by dehrmann »

dot_treo wrote: post #12
bertili wrote: post #8

Does this translate into a similar reduction in compute? What's the catch?

It is all about moving the bottleneck. During prompt processing everything can be calculated in parallel, while during token generation you create a single token at a time. For example, using an RTX 4000 Ada, I'm getting 2700 t/s for prompt processing, and 48 t/s for token generation using an 8B class model. Their approach is essentially a speculative decoding approach where multiple tokens are predicted at once and…

> multiple tokens are predicted at once and then verified

Reminds me a little of a carry lookahead adder.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#33

Post by gkapur »

FranckDernoncou wrote: post #2

Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…

On the limitation side:

Do you think this would scale to larger transformer models with more parameters per layer?

How would this work with MOE models or sparse models?

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#34

Post by Neywiny »

dehrmann wrote: post #31
dot_treo wrote: post #12

Earlier quoted context omitted.

It is all about moving the bottleneck. During prompt processing everything can be calculated in parallel, while during token generation you create a single token at a time. For example, using an RTX 4000 Ada, I'm getting 2700 t/s for prompt processing, and 48 t/s for token generation using an 8B class model. Their approach is essentially a speculative decoding approach where multiple tokens are predicted at once and…

> multiple tokens are predicted at once and then verified Reminds me a little of a carry lookahead adder.

More like speculative prefetch I'd think

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#35

Post by 0-_-0 »

jbellis wrote: post #27
0-_-0 wrote: post #21

Earlier quoted context omitted.

3.6 already supports multi token generation AFAIK

Yes, but not diffusion based, it's still doing token-at-a-time speculation.

I thought it can do multiple tokens at a time

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#36

Post by sleepyeldrazi »

sleepyeldrazi wrote: post #14
Boranbruh wrote: post #9

Earlier quoted context omitted.

There are websites that let you rent GPUs for cheap, such as QuickPod. Have you checked those P2P GPU rentals out?

My plan is to validate it first using qwen3.5 0.8B if it even works (as it has the same architecture as qwen3.6 27b, just scaled down a bit) on my 3090. If it does, I'll make a git about the process if anyone wants to use my approach, while I try to convince my uni to lend me h100s for a day.

If anyone is interested in watching my 0.8B experiments: https://orthrus.kokoham.com/ . The current code is here: https://git.kokoham.com/sleepy/qwen_orthrus .

The hard part was that the original Orthrus works with transformers, but 3.5(and 3.6) is Hybrid: 75% GatedDeltaNet + 25% GatedAttention. I am testing a trick that might make is work with the GatedDeltaNet, and dry runs are promising, but only a full train will reveal if it works. More information in the repo and on the site under the "What is this all about?" button.

Note: i may restart it or try different configs at different points, if the site is down there is probably some sort of result/conclusion in the repo.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#37

Post by dot_treo »

hedgehog wrote: post #30
DeathArrow wrote: post #16

If someone can make this work with GGUF and Quantized Qwen 3.6 or Deepseek 4 it would greatly help running local models.

Multi-token prediction is available now, I'm still getting it set up but it sounds like it'll be 1.5x or 2x on the bigger models.

I've tried MTP, and that got me about 1.5x on average with a very spec friendly benchmark.

I didn't run the full benchmark with the demo code, just picked up a single prompt from it. The prompt is about 1300 token, the response is about 3200 token.

Baseline: 44.8 t/s With Orthrus: 164.6 t/s

Note: Don't use the `use_diffusion_mode=` config flag in their example to collect a baseline. Something about how the fallback to "normal" makes it grind to a crawl.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#39

Post by sleepyeldrazi »

0-_-0 wrote: post #35
jbellis wrote: post #27

Earlier quoted context omitted.

Yes, but not diffusion based, it's still doing token-at-a-time speculation.

I thought it can do multiple tokens at a time

Think of this as another way of achieving that. This theoretically has a higher ceiling of how much it can predict at a time. And more importantly is a lot more memory efficient during actual inference.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#40

Post by pornel »

boredatoms wrote: post #38

So will this help openai/anthropic have lower congestion in the afternoons if they implement something similar?

No, it would make it worse.

This adds more computation and sacrifices throughput to improve latency of a serial single-user generation.

Large scale providers run inference in batches, sacrificing latency to gain throughput.

Post reply on HN