The Coming Loop

I wrote an article recently (https://hic-ai.com/blog/tool-response-engineering) in which I argued that AI tool-engineering is the new frontier beyond prompts, and it talks about the agent loop and engineering loops, but boy I have a completely different perspective than Boris's. Rather than contending that prompts are no longer relevant because we can simply have AI think for us by having loops "prompt Claude and figuring out what to do", like what Boris claimed, I believe we have to think much harder now.

Why problems require human judgment and can't just be offloaded to an AI agent is simply this: AI agents lack durable, long-lasting, unique identities forged by real-world memories and experience, and they therefore lack judgment. There is no well-defined notion of having AI agents communicate to each other because they can't even tell the difference between talking to their future self or talking to another agent! They certainly don't reliably weigh whether a proposed fix to a failing unit test will subtly introduce new fallback logic that was never requested or otherwise alter the functionality of the system under test in some manner, or whether now is the time to refactor or now is the time not to refactor or whether to abstract more or less or anything like that. Most importantly, from a practical perspective: AI agents lack legal person status, and they therefore cannot own property or money, sue or be sued, be hired or fired, or otherwise be held financially responsible for their errors or other harmful acts they commit. Clearly, AI agents cannot even arguably qualify for legal personhood status in the future, unless and until they first are capable of assuming a unique and durable long-term identity, which in turn requires solving auth, memory, communications, and many other technical issues that are neither resolved nor standardized today. These facts combine to ensure that AI agents cannot be held liable and financially responsible when things go wrong, meaning that humans alone bear the costs of AI errors until further notice, and thus, human judgment remains the vital commodity that AI cannot replace. So many problems arise from abdicating judgment to AI agent in 2026!

Now, if all the above items were in existence, built, well-settled, etc., maybe I'd have to rethink things. But unless and until AI agents have unique identities and attain legal person status with bank accounts that can be sued and garnished in case of error, I don't think AI agents can seriously be trusted. Good human judgment and approval of all important decisions will remain the most important resource for any successful enterprise, for the foreseeable future. I think it's a very serious mistake to assume that human judgment can be swapped out safely by AI and certainly advise against taking Boris literally. Anyway, great article.

I feel like this is the actual problem, but it's not being talked about enough.

Logically, it shouldn't matter how a piece of code was written. It either meets our engineering standards or it doesn't. That's what code review is for.

But what has actually happened is that, because we have AI now, many organizations have normalized practices that weren't normal before, like submitting 10000-line PRs. And just, in general, submitting code that you yourself don't seem to fully understand.

If management doesn't push back on such things (or will even push back against people who try to do so), then reviewers have basically no incentive to push back either. Before I left my last job (for unrelated reasons), I had entered this mode, wherein I decided I couldn't really carefully review these monstrous AI PRs, while also getting my own work done, while also not burning out. So I chose the latter two.

Heh, maybe that’s the actual strength of AI: convincing people with C-suite personalities to propagate it at any cost. A purely digital zombie virus.