Semantic unit testing: test code without executing it

From my testing the robots seem to 'understand' the code more than just learn how do thing X in code from reading code about doing X. I've thrown research papers at them and they just 'get' what needs to be done to take the idea and implement it as a library or whatever. Or, what has become my favorite activity of late, give them some code and ask them how they would make it better -- then take that and split it up into simpler tasks because they get confused it you ask them to do too much at one time.

As for debugging, they're not so good at that. Some debugging they can figure out but if they need to do something simple, like counting how far away item A is from item B, then I've found you pretty much have to do that for them. Don't get me wrong, they've found some pretty deep bugs I would have spend a bunch of time tracking down in gdb, so they aren't completely worthless but I have definitely given up on the idea that I can just tell them the problem and they get to work fixing it though.

And, yeah, they're good at writing tests. I usually work on python C modules and my typical testing is playing with it in the repl but my current project is getting fully tested at the C level before I have gotten around to the python wrapper code.

Overall its been pretty productive using the robots, code is being written I wouldn't have spent the time working on, unit testing is being used to make sure they don't break anything as the project progresses and the codebase is being kept pretty sound because I know enough to see when they're going off the rails as they often do.

Does this buy carbon offsets, too?

No, Rice's theorem states that there is no general procedure to take an arbitrary program and decide nontrivial properties of its behaviour. As software engineers, though, we write specific programs which have properties which can be decided, perhaps by reasoning specific to the program. (That's, like, the whole point of software engineering: you can't claim to have solved a problem if you wrote a program such that it's undecidable whether it solved the problem.)

The "several times more effort to verify code" thing: I'm hoping the next few generations of LLMs will be able to do this properly! Imagine if you were writing in a dependently typed language, and you wrote your test as simply a theorem, and used a very competent LLM (perhaps with other program search techniques; who knows) to fill in the proof, which nobody will never read. Seems like a natural end state of the OP: more compute may relax the constraints on writing software whose behaviour is formally verifiable.

> swap the numerator and divisor

Even Rust can express this; you don't need to get fancy. Morally speaking, division takes a Num and a std::num::NonZero.

Interesting experiment. I like that you framed it as “tests that read the docs” rather than “AI will magically find bugs”, because the former is exactly where LLMs shine: cross‑checking natural‑language intent with code.

A couple of thoughts after playing with a similar idea in private repos:

Token pressure is the real ceiling. Even moderately sized modules explode past 32k tokens once you inline dependencies and long docstrings. Chunking by call‑graph depth helps, but at some point you need aggressive summarization or cropping, otherwise you burn GPU time on boilerplate.

False confidence is worse than no test. LLMs love to pass your suite when the code and docstring are both wrong in the same way. I mitigated this by flipping the prompt: ask the model to propose three subtle, realistic bugs first, then check the implementation for each. The adversarial stance lowered the “looks good to me” rate.

Structured outputs let you fuse with traditional tests. If the model says passed: false, emit a property‑based test via Hypothesis that tries to hit the reasoning path it complained about. That way a human can reproduce the failure locally without a model in the loop.

Security review angle. LLM can spot obvious injection risks or unsafe eval calls even before SAST kicks in. Semantic tests that flag any use of exec, subprocess, or bare SQL are surprisingly helpful.

CI ergonomics. Running suite on pull requests only for files that changed keeps latency and costs sane. We cache model responses keyed by file hash so re‑runs are basically free.

Overall I would not drop my pytest corpus, but I would keep an async “semantic diff” bot around to yell when a quick refactor drifts away from the docstring. That feels like the sweet spot today.

P.S. If you want a local setup, Mistral‑7B‑Instruct via Ollama is plenty smart for doc/code mismatch checks and fits on a MacBook