51–60 of 65 posts
Earlier quoted context omitted.
Sure, but AI is trained on the code that got us here isn't it? Do you think that we're going to use less RAM with AI produced code?
"Do not use react. Do not use tailwind. We will not deploy on vercel." and suddenly things work again, it's magic!
LLM code past a few pages is really unreadable. A workflow of LLM writes code, human reads/checks/corrects is IME very unproductive and frustrating, and you're usually better off writing by hand (YMMV).
If you want to reap some productivity improvements, it can't rely on you reading, grokkong and accepting the LLM code line by line. You just kind of need to suck it in.
So the best you can do is to indeed, constrain it heavily, maybe write the interface, then have the LLM write tests, then have it write the code.
I'm not saying I like the approach, but I agree manually verifying LLM code is very frustrating and unproductive.
I keep posting this but it keeps being relevant. I had an agent implement a feature completely backwards. It wrote a whole bunch of tests proving the correctness of the implementation. All the tests passed. The really interesting thing to me is that formal verification wouldn't have helped there either -- it would have just written a mathematical proof of the correctness of the backwards feature.
A few models ago it was not unusual to find that Claude had written tests whose assertions looked correct, but were so thoroughly mocked out that they ran no real code at all — a Potemkin test. I haven’t seen it do that in quite a while, but it was an interesting failure mode!
I keep posting this but it keeps being relevant. I had an agent implement a feature completely backwards. It wrote a whole bunch of tests proving the correctness of the implementation. All the tests passed. The really interesting thing to me is that formal verification wouldn't have helped there either -- it would have just written a mathematical proof of the correctness of the backwards feature.
What do you mean by "completely backwards"?
The point was that if your bullet hits someone, you would report it to the server, and then the server would check if you got the hit or not. (So you couldn't just send fake hit reports.) So the game would still be fair and CPU usage would be greatly reduced.
The way the agent implemented it was the other way around. It had players send reports when they had been hit. So they could simply not send them and make themselves invincible. Also the game still checked collisions anyway making the whole feature pointless (on top of broken).
(Actually it made the game slower than it was originally!)
Hope that clears things up.
--
An AI agent actually pointed out the absurdity of the new code to me. So I'm assuming it wasn't that one which wrote it. Probably one of the smaller ones I was testing. But unfortunately I didn't get them to sign their names so I'll never know!
I keep posting this but it keeps being relevant. I had an agent implement a feature completely backwards. It wrote a whole bunch of tests proving the correctness of the implementation. All the tests passed. The really interesting thing to me is that formal verification wouldn't have helped there either -- it would have just written a mathematical proof of the correctness of the backwards feature.
Wait, did you let the same agent write the tests that wrote the implementation? Or was it that the plan was the wrong way round, and both the generator of code and the generator of the tests followed that same incorrect plan? Also, what were your mechanisms for reviewing the plan against the described business outcome? Anything else you could have done there to catch the backwardness? And finally, did you run any of…
In this particular instance I'm not sure what happened. I don't know where it got the idea from to do it backwards.
My best guess is that the to-do items were too vague. I built up a lot of context in my head from back and forth with the agents, and so my mental model was pretty solid, but probably an insufficient amount of that was encoded in the repo itself.
I'm not sure what you mean by reviewing the plan — do you mean making a detailed implementation plan before beginning the work?
Most of the changes were pretty straightforward, or at least we'd already worked out most of the details and put them into to-dos. So for the most part my prompt was just "alright go ahead and implement the next thing on the list."
If I had to guess I'd say the main reason it went wrong was because the why was missing. The to-do item specified what work remained to be done, but it did not explain the reason for each item. I think that was the core of the issue.
So it probably ended up seeing a bunch of individual changes out of context and not understanding what the point was supposed to be.
---
>did you run any of it across models from different foundation labs?
Working on a different part of the same repo a week later, I used Fable and Sol to find issues. Then I let them run cross critique on each other's reports. Then I had each of them generate a plan, and then I had them do cross critique on the plans. And then I repeated that until they were both satisfied with the results, i.e. until the two plans merged into one coherent plan.
It was interesting because they're both had different strengths and weaknesses in different parts of the process. (One of them sound more issues in the initial phase, but the other came up with a more comprehensive fix for each one.)
I've seen very promising results for model alloys with security research (it was posted here about a year ago[0]), so I wanted to give it a try myself.
There doesn't seem to be a very convenient way to do it (maybe one of the new harnesses which runs the proprietary harnesses as subprocesses?), so I just passed markdown files between the two models manually.
"What kind of software do you have agents writing without human supervision?"