Viewing profile — languid-photic

also feels like a good posttraining task

would be fun to do zig -> rust -> zig and to measure the delta (in a VAE-ish way, kl div on the embeddings?)

It’s mostly a bandwidth thing. We’ve seen the pattern consistently, but haven’t had time yet to write up the analysis carefully. We are not the only ones to see the reasoning inver…

My point is more reasoning often leads to worse "scope creep/churn, codebase fit, maintainability".

We also use a secondary signal from blinded multi-verifier reviews. Each verifier ranks the candidates, and those verification outcomes serves as an additional quality signal. It's…

Agreed. Harness is really important. Especially since many labs are now post-training agents directly in their native harness. (Which is why my prior is that third party harnesses …

Yes, the signal we are measuring is quite different from most evals. We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patc…

I agree! So far we have been native harnessmaxxing, which simplifies things a lot. The configuration space around open models is much larger. Eg which models, capability heterogene…

Yes! It depends on the extent of changes needed. If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly. If the changes …

We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, dif…

Agreed. As alignment improves, I'm becoming increasingly bearish on sandboxing. Version control and isolation will probably stay useful, though, more for distributed development an…

It’s very hard to encode the properties that matter most in code in tests. [1] [1] https://voratiq.com/blog/your-workflow-is-the-eval

I think that's Pro. Regular 5.5 is 2x regular 5.4.

It's 2x/token, but for default reasoning we've found GPT-5.5 uses fewer tokens overall, so net cheaper on median. [1] (Note, that stops being true at higher reasoning levels, where…

Appreciate the clarification. But, it's still not great. To the PM behind this - developers are sensitive to this kind of thing. Just make it opt-in instead?

it's reasonable to note that w/o sharing the data these findings can't be audited or built upon but i think the prior on 'this team fabricated these findings' is v low

makes sense! we wrote something yesterday about the weaknesses of test-based evals like swe-bench [1] they are definitely useful but they miss the things that are hard to encode in…

Yes! The spec is the source of truth. Voratiq converts it into a canonical prompt so every agent gets identical instructions.

Good point. This post measures `1x top-N` (one attempt each from N models), not `Nx top-1` (N attempts from the best-scoring model). We should make that more clear. Part of why we …

It was not, the agent id is not overt but can be found via the workspace filepath. But that is a good point. Perhaps it should be mapped to something unidentifiable.

https://github.com/voratiq/voratiq For comparison, there's a `review` command that launches a sandboxed agent to review a given run and rank the various implementations. We usually…

Yes, understandable. The question is which multi-agent architecture, hierarchical or competitive, yields the best results under some task/time/cost constraints. In general, our sen…

Yes indeed, you get a big lift out of running just the few top agents. We run big ensembles because we are doing a lot of analysis over the system etc

This was exactly the kernel of the idea :)