31–40 of 75 posts
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
The future might be parallel attempts and selecting the best-of-N solutions.
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
So - apparently it's not fully self-hosted, since I don't see a GPU. I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
Some people have alternative models review each other's output. The future might be parallel attempts and selecting the best-of-N solutions.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions. Wondering what the actual verification loop looks like once you start taking these systems to production.
For every property of the system, implement a mock that simulates a system that works, and multiple variations with forseeable bugs (models are fairly good at predicting possible bugs). Then, write a unit tests that pass on the working simulation and fail on the buggy simulations.
This way model ensures reasonably well that the unit tests test what they're supposed to. Then, you use those tests as a verification to guide the model to implement the functionality.
This approach relies on the assumption that the model can design a reasonable architecture composed of independently-testable modules. In my experience, frontier models should be able to do that.