After two years of vibecoding, I'm back to writing by hand

I get the sense that many programmers resent writing tests and see them as a checkbox item or even boilerplate, not a core part of their codebase. Writing great tests takes a lot of thought about the myriad of bizarre and interesting ways your code will run. I can’t imagine that prompting an LLM to “write tests for this code” will result in anything but the most trivial of smoke test suites.

Incidentally, I wonder if anyone has used LLMs to generate complex test scenarios described in prose, e.g. “write a test where thread 1 calls foo, then before hitting block X, thread 2 calls bar, then foo returns, then bar returns” or "write a test where the first network call Framework.foo makes returns response X, but the second call returns error Y, and ensure the daemon runs the appropriate mitigation code and clears/updates database state." How would they perform in this scenario? Would they add the appropriate shims, semaphores, test injection points, etc.?

Right, and he also very recently said that he felt essentially left behind by AI coding advances, thinking that his productivity could be 10x if he knew how to use it better.

It seems clear that Karpathy himself is well aware of the difference between "vibe coding" as he defined it (which he explicitly said was for playing with on hobby projects), and more controlled productive use of AI for coding, which has either eluded him, or maybe his expectations are too high and (although it would be surprising) he has not realized the difference between the types of application where people are finding it useful, and use cases like his own that do not play to its strength.

One use case that I'm beginning to find useful is to go into a specific directory of code that I have written and am working on, and ask the AI agent (Claude Code in my case) "Please find and list possible bugs in the code in this directory."

Then, I can reason through the AI agent's responses and decide what if anything I need to do about them.

I just did this for one project so far, but got surprisingly useful results.

It turns out that the possible bugs identified by the AI tool were not bugs based on the larger context of the code as it exists right now. For example, it found a function that returns a pointer, and it may return NULL. Call sites were not checking for a NULL return value. The code in its current state could never in fact return a NULL value. However, future-proofing this code, it would be good practice to check for this case in the call sites.

"I really hate this damn machine. I wish that they would sell it. It never does what I want it to, only what I tell it."

On the one hand, I created vibe coded a large-ish (100k LOC) C#, Python, Powershell project over the holidays. The whole thing was more than I could ever complete on my own in the 5 days it took to vibe code using three agents. I wrote countless markdown 'spec' files, etc.

The result stunned everyone I work with. I would never in a million years put this code on Github for others. It's terrible code for a myriad reasons.

My lived experience was... the task was accomplished but not in a sustainable way over the course of perhaps 80 individual sessions with the longest being multiple solid 45 minute refactors...(codex-max)

About those. One of things I spotted fairly quickly was the tendency of models to duplicate effort or take convoluted approaches to patch in behaviors. To get around this, I would every so often take the entire codebase, send it to Gemini-3 Pro and ask it for improvements. Comically, every time, Gemini-3-Pro responds with "well this code is hot garbage, you need to refactor these 20 things". Meanwhile, I'm side-eying like.. dude you wrote this. Never fails to amuse me.

So, in the end, the project was delivered, was pretty cool, had 5x more features than I would have implemented myself and once I got into a groove -- I was able to reduce the garbage through constant refactors from large code reviews. Net Positive experience on a project that had zero commercial value and zero risk to customers.

But on the other hand...

I spend a week troubleshooting a subtle resource leak (C#) on a commercial project that was introduced during a vibe-coding session where a new animation system was added and somehow added a bug that caused a hard crash on re-entering a planet scene.

The bug caused an all-stop and a week of lost effort. Countless AI Agent sessions circularly trying to review and resolve it. Countless human hours of testing and banging heads against monitors.

In the end, on the maybe random 10th pass using Gemini-3-Pro it provided a hint that was enough to find the issue.

This was a monumental fail and if game studios are using LLMs, good god, the future of buggy mess releases is only going to get worse.

I would summarize this experience as lots of amazement and new feature velocity. A little too loose with commits (too much entanglement to easily unwind later) and ultimately a negative experience.

A classic Agentic AI experience. 50% Amazing, 50% WTF.