Earlier quoted context omitted.
Context is the most challenging bit. FWIW, the codebases I'm working on are still small enough to where I rarely need to include more than 12 files into context. And I find as I make the context bigger beyond that, results degrade significantly. So I don't know how this would go in a much larger codebase. What floored him was simply how much of my programming I was doing with an LLM / how little I write line-by-line…
So you spend 10 minutes writing a free text description of the test you want; tell it exactly how you want it to write the test, and then 4-5 minutes trying to understand if it did the right thing or not, restart because it did something crazy then a few minutes manually fixing the diff it generated? MMmm. I mean, don't get me wrong; this is impressive stuff; but it needs to be an order of magnitude less 'screwing ar…
You summed up the workflow accurately. Except, I read your first paragraph in a positive light, while I imagine you meant it to be negative.
Note the feedback loop you described is the same one as me delegating requirements to someone else (i.e. s/LLM/jr eng). And then reading/editing their PR. Except the feedback loop is, obviously, much tighter.
I've written a lot of tests, I think this would have taken 3-4x longer to do by hand. Surely an hour?
But even if all things were roughly equal, I like being in the navigator seat vs the driver seat. Editor vs writer. It helps me keep the big picture in mind, focused on requirements and architecture, not line-wise implementation details.