Viewing profile — Edward40

[flagged]

We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.

I also prefer using vanilla Pi over Oh My Pi.

Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complet…

Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This…

We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.

Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interac…

The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.