21–30 of 37 posts
This has got to be the worst project name I've seen all year.
I think the name probably has to do with this research from Microsoft (same people involved and a lot of similarities in the way it's conceived. HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools : https://arxiv.org/pdf/2605.17106
But could be your thing too obviously.
> One model drafts a result, an independent read-only critic from a different model family reviews it Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does. Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor. The ablations [1] p…
My editor and gatekeeper use like the same model Opus 5. Different prompts and a kind of different input data. The gatekeeper receives the fact check results next to finished text. In the same time the editor already delivered them. As far as I remember over the entire period he removed 27 posts out of 187 that went through him. So I believe that different manufacturers are not mandatory. What matters I guess is not…
Different inputs are one point, but there is another problem: lack of diversity
Models from the same maker, share the same training and the same implicit bias. It is like if both reviewers had the same gender, race, and studied at the same university, and just got different book day before. Add fresh immigrant from rural asia, you get VERY different opinions, even with the same input book...
Plus practical aspects, Opus 5 is sometimes way too creative which is good for writting. GPT Sol is complete oposite, it is obsessed with crossing every T and verifying every dot. It complements Opus as reviewer!
If opus gets security sensitive questions, gets downgraded to sonnet and againdown to haiku, the same model will hit the same security block, and will not catch the issue. Model from another lab will very likely catch this.
Plus anthropic models love to smell their own farts, load bearing seems are fantastic...
I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness. This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair. Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Termina…
> Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics Do you have a link/paper for this you could share?
In fact that is how I got the idea. There is a 4 month old post on SWEbench github that claimed 20 point boost (beating opus as well) for another small Qwen model when it's traffic was sent via a similar proxy and claude code was used as the harness.
The author didn't post their full code, but he did post all sorts of logs showing which features of his proxy were activated during which run.
I wanted to exclude the harness and validate his claims regarding only the proxy's impact so I reimplemplemented a subset of his features that seemed to give me the biggest "wins" with the smallest amount of work. These are the ones I list in the parent.
It took one afternoon last week plus the runs that took few days.
I saw about 10% better results.
As I mentioned, it is a subset of tasks, not the entire set, but results so far are very encouraging.
Since then I implemented many more features in the proxy (best of N for example), but i haven't had time to measure the impact yet. His proxy also had best of N, but that feature wasn't activated in on the runs he reports.
I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness. This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair. Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Termina…
Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.
I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doesn't need it - or it's already done behind the scenes.
Earlier quoted context omitted.
Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.
Earlier quoted context omitted.
Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
It is useful to compare like for like. Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models. I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doe…
I think the benchmarks should be trying to use realistic harnesses for both proprietary and open weight models.
Earlier quoted context omitted.
Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.
I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the m…
It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.
> One model drafts a result, an independent read-only critic from a different model family reviews it Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does. Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor. The ablations [1] p…