Measuring the sloppiness of code

I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.

Plopping in my email to the author below in case anyone else is interested in this kind of thing:

> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com

Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".

In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.

Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:

L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.

Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.

I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.

I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.

Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.

Best, Michael

PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20

This doesn’t scale at all for some problems where a specific algorithm is needed (graph problems for instance).

I'm pretty sure we reached peak software developer jobs due to ai.

The market is already collapsing in China and India.

I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.