The Speed of Prototyping in the Age of AI

Now the quality of what comes after the iteration depends on the reviewer, and that's no AI's fault.

And I don't doubt you have had you agro with LLMs, because I've also had my fair share of issues with them, I just think we have different emotional responses to the workflow with agents. They don't work the first time and they aren't very good at sweeping large sets of loosely related changes. They need to focus on one feature only and crunch it to the end.

Honestly though I've didn't have the chance to work in large codebases, but with those projects I had lots of success and I found the workflow very stimulating, reading the solutions the LLM come up with, some very interesting and some comically bad, but more often than not I'll pick up a technique or an approach I didn't think about. Worse case it's something I can bounce ideas off of.

About bugs, I have the opposite impression. I find it really interesting to get a functionality wrong, provide the agent with the logs and context and explain in detail the issue and have it help me explore the codebase to identify and fix the issue. I've never had an instance until now that I couldn't fix the bug or that I left the session in a worse mental state than I entered.

I'll take buzz, for instance. Before using zurg I had to use Plex because jellyfin would only detect a single file in a folder with multiple files. Codex created the presentation layer I described in a single go and it worked first time. That was really impressive I have to say. The project also has it's own WebDAV server, it integrated with debrid, has a persistent catalogue of media that is independent of debrid and can be used to restore previously deleted media. It has a logging UI, a config UI and a nice event system for waiting for different independent services that it needs to orchestrate. I don't think it's a large codebase, but it's nowhere near a toy project. It also has a very capable CI pipeline that supports the development. The only part I couldn't get the agent to do well for nothing was frontend implementation, maybe because I refused to use a framework and defaulted to plain JavaScript and CSS embedded in jinja2 templated html files. I have picked up a couple of techniques when I did full stack work when I was an intern so I was cabaple of using the browser to inspect and refine the Dom elements. One thing that it did poorly for instance was to create all elements in block display, however planning a refactor to use flexbox throughout the code really improved the UI resilience and it was really effortless to deploy. In buzz I haven't touch most of the code, just some adjustments in the htmls to serve as an example for the agent of how to do it correctly, prompts not being the only way to interact with them, but I read most of the code and validated most of the functionality in merge requests, just like you'd do in a team work.

In a nutshell I think agents are really capable since November last year of working in large code bases, but I don't trust them to just be let loose. They need lots of hand holding and steering, but for me once I got the hang of it I really feel like I'm extremely productive.

My hypothesis is that people are more likely to have success with agents the more they enjoy writing in natural language and reading code, while people that prefer coding and dislike writing text will usually prefer handcrafting their programs.