Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to…
> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would nee…
I'm curious whether this is something LLMs are eventually going to be good enough at doing, or something the average developer knows when and how to do, or if this is going be something that's too specialized or difficult for most developers and maybe the next generation of developer tooling products. Now that we're several months into Claude Code crossing the threshold of legitimacy and adoption, I've been surprised at how few projects or developers are doing this yet.
To a certain extent now all you need to do is ask Claude Code for browser automation workflows and CUJ tests in your repo, and ye shall receive, but probably something that just uses base playwright. It would be even better if you could ask to install or use a self-eval tool that already did everything you needed it to do and also knew how to specify/setup automations. I'm assuming the level of agency or mental overhead of embarking on a browser automation side quest is beyond what most developers are used to in the course of their regular work, even though it's not really as hard as it sounds now. If so then self-eval tooling could be a very promising new product category to sell to enterprises.
BTW if you have a link to your project I'd be interested in checking it out! $5.69 for a UAT run sounds very high to me based on how many tokens it typically takes for agents to create automations or steer my similar project, but it could be that your test workloads are much more exhaustive or high-dimensionality than mine are. This is what a basic "go to amazon.com and search for a product, then take a screenshot" automation looks like for ours: https://github.com/accretional/chromerpc/blob/main/recipes/s.... And this is our interactive/dynamic remote steering mode: https://github.com/accretional/chromerpc/blob/main/chrome-pr.... I decided to implement against the Chrome Devtools Protocol (one layer under Playwright) and use grpc service reflection to allow agents to dynamically discover/describe the entire chrome devtools api surface. I just started working on a way to gather traces and monitor/manage the automation run internals because I think there's a ton of opportunity in this problem space for orchestration and RL