> But today I had some exposure to a SotA agentic team coding loop thingie which had been running almost hands-off for a few weeks on a (pretty serious) Win32+Direct3D-to-Emscripten+WebGL porting project - and I'm genuinely spooked at how well it all works; I mention this example because all the agents' processes involved a decently rigorous verification step: any time any agent confidently asserts something then it has to provide associated evidence, such as a unit test report, or build artefact, or external citation, and the system will spawn a new agent (perhaps using a different backing LLM) to verify the claim. I know a unit-test pass/fail isn't quite the same thing as, say, a medical AI agent confidently wrong about me having/not-having terminal spleen cancer, but the capability for a team-of-agents to be self-checking is definitely there.
It taking "a few weeks" of wall-clock runtime suggests to me that it's making a lot of mistakes that keep needing correction, you're seeing the output after it fixes them. FWIW, my experience with them includes just enough fragile self-made unit tests to be a problem for entirely hands-off work, despite what it gets right.
However, on the topic of time rather than quality: Last summer I decided to convert an old game from C++/SDL2 to JS/WebGL, and was doing that with manual copying each function one at a time to ChatGPT and then code reviewing each line by hand. The wall-clock speedup from proper agentic coding that has come since then, is so much much faster than me working like this, that the modern approach would have been done in less than a week if it had managed without making mistakes.
I might go back to that project. While I was enjoying the process of code archeology (the code being a mix of many platforms starting with 68k era classic MacOS software rendering and resource forks upwards), I had to focus on other things for a bit in the meanwhile, which is of course the actual benefit of code factories.