Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Openrouter Fusion API

openrouter.ai

Post reply on HN
Print view View original

41–50 of 98 posts

Re: Openrouter Fusion API

#41

Post by dist-epoch »

dsl wrote: post #16

Heh. I built "Fusion" a few months ago as an MCP using OpenRouter. The idea was to give Claude a "panel of experts" to go talk to when it got stuck. After extensive testing and benchmarking I discovered that when you ask one model to judge another's response you don't actually get a better answer. You are just asking it "how closely does this resemble the answer you would have given me." Additional rounds and all the…

I think it depends.

I regularly ask both GPT and Gemini to give me options - programming libraries to do X, architecture suggestions, names for projects/services/classes

After they answer I ask each model what does it think of the other answer, and to give me a final suggestion considering both answers.

Both GPT and Gemini would frequently say "that other answer is much better than my one, it considered X factor that I missed".

Re: Openrouter Fusion API

#42

Post by efromvt »

dsl wrote: post #16

Heh. I built "Fusion" a few months ago as an MCP using OpenRouter. The idea was to give Claude a "panel of experts" to go talk to when it got stuck. After extensive testing and benchmarking I discovered that when you ask one model to judge another's response you don't actually get a better answer. You are just asking it "how closely does this resemble the answer you would have given me." Additional rounds and all the…

I’d be interested in the benchmarking if you ever write it up! People do seem to assume LLM as a judge/panel improves outcomes (and arguably it does in cases like code review?) but I suspect it is very situational and the priors from human panel of experts don’t always translate cleanly.

Re: Openrouter Fusion API

#43

Post by alex7o »

I have been thinking a lot about this and my simplified understanding is that each model can be seen as a bell curve over human knowledge and each model has a different distribution. Using multiple models would allow us to change the distribution of other models with text that is out of their original curve. But then if you think about it does SFP and RL even alter the original distribution of text enough that models have enough variety so that their combined output is something better or just an echo chamber I believe not but I have no way to prove it yet.

Re: Openrouter Fusion API

#44

Post by andai »

sigmoid10 wrote: post #4
andai wrote: post #2

Context: Surpassing Frontier Performance with Fusion https://news.ycombinator.com/item?id=48525392 And a slightly better UI here: https://openrouter.ai/fusion On OpenRouter's fusion API your request is routed to several models simultaneously and a judge model combines their answers into a final response. This significantly boosts performance, at the cost of time (at least on the one benchmark they tested, a deep rese…

Interesting how well a panel of Fable 5 + GPT 5.5 beats the frontier of either one, but if you add Gemini into the mix the panel of three performs worse, not better. To me that sounds like Gemini is worse at the given tasks but better at convincing judges of its solutions. Oh and a Panel of 2 Opus 4.8 models is almost exactly as good as one Fable 5. That smells suspicious. Do we know if that might simply be what Anth…

> Interesting how well a panel of Fable 5 + GPT 5.5 beats the frontier of either one, but if you add Gemini into the mix the panel of three performs worse

I'm not seeing that? Did you maybe misread the #2 ranked one as Fable + GPT + Gemini? It's actually Opus + GPT + Gemini.

Re: Openrouter Fusion API

#45

Post by sfilkin »

dsl wrote: post #16

Heh. I built "Fusion" a few months ago as an MCP using OpenRouter. The idea was to give Claude a "panel of experts" to go talk to when it got stuck. After extensive testing and benchmarking I discovered that when you ask one model to judge another's response you don't actually get a better answer. You are just asking it "how closely does this resemble the answer you would have given me." Additional rounds and all the…

I've been thinking along those lines, too. Could you give a general overview of your solution?

Re: Openrouter Fusion API

#46

Post by andai »

qsort wrote: post #6
sigmoid10 wrote: post #4

Earlier quoted context omitted.

Interesting how well a panel of Fable 5 + GPT 5.5 beats the frontier of either one, but if you add Gemini into the mix the panel of three performs worse, not better. To me that sounds like Gemini is worse at the given tasks but better at convincing judges of its solutions. Oh and a Panel of 2 Opus 4.8 models is almost exactly as good as one Fable 5. That smells suspicious. Do we know if that might simply be what Anth…

Yeah, GPT 5.5 + Fable beating either individually is belivable, but 2x Opus > Fable is what makes me a bit dubious about the whole thing. They might be measuring skills that are too specific or benefit a lot from more tokens being thrown at them. Also Claude Code (the harness) is not the best at the moment, that might be part of it as well?

What throws me off is DeepSeek beating both Opus 4.8 and GPT 5.5.

That definitely doesn't sound right.

Re: Openrouter Fusion API

#47

Post by bsenftner »

Oras wrote: post #27
bsenftner wrote: post #26

I'm sure many have made something like this, I've done a few. I've found simply submitting one's prompt to multiple models to be kind of pointless. You're just going to get statistical noise from the variances in their training methods, as they are all training on pretty much the same data. I get significantly better results by pre-prompting each LLM (they can be the same LLM too, just another instance), I pre-prompt…

Agree, and I see opus and Gemini pro as “quality” on openrouter fusion, this would be super pricy if the prompts are dynamic and not optimised for caching. I would love to hear why they have created it, what was the business case, what this is going to serve? As you said, this is pretty easy to replicate

[dead]

Re: Openrouter Fusion API

#49

Post by andai »

jorvi wrote: post #5
andai wrote: post #2

Context: Surpassing Frontier Performance with Fusion https://news.ycombinator.com/item?id=48525392 And a slightly better UI here: https://openrouter.ai/fusion On OpenRouter's fusion API your request is routed to several models simultaneously and a judge model combines their answers into a final response. This significantly boosts performance, at the cost of time (at least on the one benchmark they tested, a deep rese…

I don't know if it is still the case with current models, but a few generations back Microsoft had some research results where asking a model to iterate N times would significantly improve performance, with the optimal point being 4 iterations.

I think there's a sweet spot for it. If a model can't do a task, iterating won't help. If a model can do it reliably, there's no need to iterate.

If it can do it, but unreliably, that's where you would get major gains from iterating. I think the Chinese models are in that sweet spot, for many tasks. I would love to test that.

I started working on my own fusion system yesterday. I'm not sure how to benchmark it though.

The thing I'm most interested in is reliability. Going from 90% to 95% on a benchmark doesn't seem like much but you've cut the error rate in half.

Post reply on HN