Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project HydraFusion: Frontier quality via multi-model orchestration

github.blog

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#11

Post by guybedo »

i've been using adversarial critique and reviews for many planning, solution design and implementation steps inside workflows.

It's so effective and helps catching so many design flaws, implementations misses etc ... that i'm wondering how people manage to build complex/large projects with agents without this kind of process. Well, i actually built this thing because i couldn't get good results so i had to find a way.

I'm gonna open source the whole thing but it needs some cleanup, there's a basic landing page here https://kodfactory.com if anyone wants to be notified when it's released on github. Yeah i know, the world really needs another software factory :-)

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#12

Post by K3UL »

zatkin wrote: post #2

This has got to be the worst project name I've seen all year.

I think the name probably has to do with this research from Microsoft (same people involved and a lot of similarities in the way it's conceived.

HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools : https://arxiv.org/pdf/2605.17106

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#14

Post by Roark66 »

I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.

This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.

Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics: - tweaks few decode settings like slightly higher temperature - detects when model gets stuck and tells it to "go on" - detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"

And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#15

Post by K3UL »

ValentineC wrote: post #8

From the article: > In controlled offline evaluations, HydraFusion’s selective coding workflows matched or exceeded the evaluated Opus 5 baseline Opus 5 (in practice) is not a good baseline to compare against. I'm shocked that they chose to primarily compare against Opus 5 in all the article's charts. It's pretty disingenuous that they're claiming "frontier" quality, but didn't compare against Fable or Sol.

I would imagine they did not test against Fable because Microsoft and GitHub (like many big companies) have internally given the instruction not to use this model, because of the data retention policy.

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#16

Post by sanderjd »

Roark66 wrote: post #14

I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness. This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair. Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Termina…

Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#18

Post by timschmidt »

sanderjd wrote: post #16
Roark66 wrote: post #14

I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness. This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair. Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Termina…

Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.

Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#19

Post by kouteiheika »

Roark66 wrote: post #14

I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness. This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair. Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Termina…

> Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics

Do you have a link/paper for this you could share?

Re: Project HydraFusion: Frontier quality via multi-model orchestration

#20

Post by soricus »

gopalv wrote: post #5

> One model drafts a result, an independent read-only critic from a different model family reviews it Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does. Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor. The ablations [1] p…

My editor and gatekeeper use like the same model Opus 5. Different prompts and a kind of different input data. The gatekeeper receives the fact check results next to finished text. In the same time the editor already delivered them. As far as I remember over the entire period he removed 27 posts out of 187 that went through him. So I believe that different manufacturers are not mandatory. What matters I guess is not the difference in models but... the fact that the critic has a different input and doesn’t have their own text that needs to be defended. That's the point
Post reply on HN