Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

github.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#41

Post by dot_treo »

sleepyeldrazi wrote: post #36
sleepyeldrazi wrote: post #14

Earlier quoted context omitted.

My plan is to validate it first using qwen3.5 0.8B if it even works (as it has the same architecture as qwen3.6 27b, just scaled down a bit) on my 3090. If it does, I'll make a git about the process if anyone wants to use my approach, while I try to convince my uni to lend me h100s for a day.

If anyone is interested in watching my 0.8B experiments: https://orthrus.kokoham.com/ . The current code is here: https://git.kokoham.com/sleepy/qwen_orthrus . The hard part was that the original Orthrus works with transformers, but 3.5(and 3.6) is Hybrid: 75% GatedDeltaNet + 25% GatedAttention. I am testing a trick that might make is work with the GatedDeltaNet, and dry runs are promising, but only a full train will…

I would probably treat the (3 GatedDeltaNet + 1 GatedAttention) Blocks as one transformer block, when generating next steps one would therefore use the kv cache for the gated attention and skip the entire delta nets.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#42

Post by regularfry »

0-_-0 wrote: post #35
jbellis wrote: post #27

Earlier quoted context omitted.

Yes, but not diffusion based, it's still doing token-at-a-time speculation.

I thought it can do multiple tokens at a time

There was a chart from the Unsloth folks posted to Reddit in the last couple of days which showed that the draft sweet spot for MTP was 2-3 tokens ahead depending on the quant. Thats not much, and I think this might do a lot better. The whole "provably identical distribution" thing is doing a lot of work in my head, and I don't think that's true of the MTP model in qwen's architecture.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#43

Post by regularfry »

dot_treo wrote: post #15
kreelman wrote: post #13

Earlier quoted context omitted.

Fantastic results. Well done. ...So this is built into the way the model works.. if I'm understanding it correctly. I was wondering what would be involved in getting it to work with GGUF files, rather than safetensor files...

Just to get it into a GGUF file would be fairly trivial. But using that GGUF file would need a bunch of additional things. One would need to create a new architecture derived from Qwen3, and then probably adapt the speculative decoding functionality. At the moment not even MTP is merged into llama.cpp, so I wouldn't quite hold my breath for it.

MTP merged today, a couple of hours after your post by the looks of things.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#44

Post by dot_treo »

sleepyeldrazi wrote: post #36
sleepyeldrazi wrote: post #14

Earlier quoted context omitted.

My plan is to validate it first using qwen3.5 0.8B if it even works (as it has the same architecture as qwen3.6 27b, just scaled down a bit) on my 3090. If it does, I'll make a git about the process if anyone wants to use my approach, while I try to convince my uni to lend me h100s for a day.

If anyone is interested in watching my 0.8B experiments: https://orthrus.kokoham.com/ . The current code is here: https://git.kokoham.com/sleepy/qwen_orthrus . The hard part was that the original Orthrus works with transformers, but 3.5(and 3.6) is Hybrid: 75% GatedDeltaNet + 25% GatedAttention. I am testing a trick that might make is work with the GatedDeltaNet, and dry runs are promising, but only a full train will…

And it also looks like the original authors are working on qwen 3.5 too: https://github.com/chiennv2000/orthrus/issues/1#issuecomment...

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#45

Post by sleepyeldrazi »

sleepyeldrazi wrote: post #36
sleepyeldrazi wrote: post #14

Earlier quoted context omitted.

My plan is to validate it first using qwen3.5 0.8B if it even works (as it has the same architecture as qwen3.6 27b, just scaled down a bit) on my 3090. If it does, I'll make a git about the process if anyone wants to use my approach, while I try to convince my uni to lend me h100s for a day.

If anyone is interested in watching my 0.8B experiments: https://orthrus.kokoham.com/ . The current code is here: https://git.kokoham.com/sleepy/qwen_orthrus . The hard part was that the original Orthrus works with transformers, but 3.5(and 3.6) is Hybrid: 75% GatedDeltaNet + 25% GatedAttention. I am testing a trick that might make is work with the GatedDeltaNet, and dry runs are promising, but only a full train will…

It is actually very exciting that they are also working on 3.5, I will keep this toy project up in the meantime, trying it out and testing things around it helps me learn a bunch.

As for the treating them as a block idea, that was my initial plan, but the GatedDeltaNet is doing most of the work in 3.5. Trying to bundle them together would hurt acceptance rates drastically, potentially making the speed benefits not a lot bigger, or smaller, than the native MTP.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#46

Post by dot_treo »

regularfry wrote: post #43
dot_treo wrote: post #15

Earlier quoted context omitted.

Just to get it into a GGUF file would be fairly trivial. But using that GGUF file would need a bunch of additional things. One would need to create a new architecture derived from Qwen3, and then probably adapt the speculative decoding functionality. At the moment not even MTP is merged into llama.cpp, so I wouldn't quite hold my breath for it.

MTP merged today, a couple of hours after your post by the looks of things.

By the looks of it, it will take a couple more follow up PRs to clean things up a bit and get the most performance from MTP. I hope that by that point it will be easier to add more spec decoding types.

In the meantime I've benchmarked Orthrus some more and got some quite promising results. So I'd be glad if my prediction that it may take some time until it lands in llama.cpp turns out to be wrong.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#47

Post by p0w3n3d »

boredatoms wrote: post #38

So will this help openai/anthropic have lower congestion in the afternoons if they implement something similar?

I hope it helps finally running large models on normal hardware. Tying our work to two companies in the world is a bad bad thing. Quite risky. Against any threat modelling

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#49

Post by [deleted] »

littlestymaar wrote: post #11
bertili wrote: post #8

Does this translate into a similar reduction in compute? What's the catch?

> Does this translate into a similar reduction in compute? No, quite the opposite actually. Like with speculative decoding this model will compute more tokens and discard the invalid ones. > What's the catch? LLMs[1] are limited by memory latency and not by compute[2]: because they process tokens one at a time, you spend more time loading and unloading the weights on the GPU registers from VRAM than waiting for compu…

[deleted]

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#50

Post by deflator »

FranckDernoncou wrote: post #2

Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…

I'm sure I don't understand all the technical aspects, but I do understand that this is frickin' cool. Nice work.
Post reply on HN