Semantic unit testing: test code without executing it

Actually writing the tests is far more effective, and doesn't require fancy frameworks tightly coupled with external services.

Broadly speaking, linters are good, and if you have a way of linting implementation errors it's probably helpful.

I would say it's probably more helpful while you're coding than at test/CI time because it will be, indubitably, flakey.

However, for a local developer workflow I can see a reasonable value in being able to go:

Take every function in my code and scan it to figure out if you think it's implemented correctly, and let me know if you spot anything that looks weird / wrong / broken. Ideally only functions that I've touched in my branch.

So... you know. Cool idea. I think it's overselling how useful it is, but hey, smash your AI into every possible thing and eventually you'll find a few modestly interesting uses for it.

This is probably a modestly interesting use case.

> suite allows you to run the tests asynchronously, and since the main bottleneck is IO (all the computations happen in a GPU in the cloud) it means that you can run your tests very fast. This is a huge advantage in comparison to standard tests, which need to be run sequentially.

uh... that said, saying that it's fast to run your functions through an LLM compared to, you know, just running tests, is a little bit strange.

I'm certain your laptop will melt if you run 500 functions in parallel through ollama gemma-3.

Running it over a network is, obviously, similarly insane.

This would also be enormously and time consuming and expensive to use with a hosted LLM api.

The 'happy path' is probably having a plugin in your IDE that scans the files you touch and then runs this in the background when you make a commit somehow using a local LLM of sufficient complexity it can be useful (gemma3 would probably work).

Kind of like having your tests in 'watch mode'; you don't expect instant feedback, but some-time-after you've done something you get a popup saying 'oh hey, are you sure you meant to return a string here..?'

Maybe it would just be annoying. You'd have to build it out properly and see. /shrug

I think it's not implausible though, that you could see something vaguely like this that was generally useful.

Probably what you see in this specific implementation is only the precursory contemplations of something actually useful though. Not really useful on its own, in its current form, imo.

I have the toughest time trying to communicate why f(x) should equal f(x) in the general case.

Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes?

I think there are also other problems (inlining a verbal description makes the codebase verbose, writing a precise, non-ambiguous verbal description might be more work than writing unit tests)