Coding might be "solved" but coding still is the best way to build your own mental model of the solution space. Which is more important to you: Velocity to a solution? Or velocity to understanding?
Velocity to a paycheck
Measuring the sloppiness of code
231–240 of 246 posts
Re: Measuring the sloppiness of code
#232The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more. Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and tr…
He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.
The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.
Re: Measuring the sloppiness of code
#233Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team. If humans increasingly are kept out of coding, then who holds the mental model? If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM spea…
We'll invent other tools to help with the mental models, ones that are more intuitive. Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected. We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operatin…
> Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
No need to imagine. Just open the README.md of many projects.
Re: Measuring the sloppiness of code
#234Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team. If humans increasingly are kept out of coding, then who holds the mental model? If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM spea…
Mental models existed before coding. The pyramids were not built by just a couple of guys hauling some rocks together. It was a highly structured organization.
Or the East India Company or many other great organizations that achieved big things before computers.
Re: Measuring the sloppiness of code
#235I just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents.
I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the complexity of code.
Re: Measuring the sloppiness of code
#236I just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents. I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the comp…
are you able to share what you're working on that's 250k LoC?
Re: Measuring the sloppiness of code
#237The most surprising indicator of messiness I encountered wasn't in the code itself, but in the tests. I had two tests that looked like tests but didn't actually measure anything—they simply returned True — and neither static analysis nor coverage tools caught them. The only thing that detected them was mutation testing—deliberately removing a line of code to see if the test fails; without that step, the test is essentially toothless. Also, I found that scores became inflated if you counted cases where a test passed despite the removal of code (because another test caught the issue) as a successful "detection."
Re: Measuring the sloppiness of code
#238Coding has been solved for 20 years at least. 90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM. 90% of code before LLMs was badly copied from StackOverflow anyway. That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to l…
>LLMs haven't created a
I don't think this is true. They very well might have, but a specific one, not a generic reusable version you find on GitHub or whatever.
Re: Measuring the sloppiness of code
#239Earlier quoted context omitted.
Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff. Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions. Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i... The underlying docket is…
Here’s the oneshot, not sure if it’s slop or not though :)
https://kolinko.eu/pdf-reading-order/
But I wonder about your opinion.
Re: Measuring the sloppiness of code
#240I just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents. I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the comp…
are you able to share what you're working on that's 250k LoC?
It's a life tracker, that has come to include everything but the kitchen sync (journal, workouts, food, Garmin stats, music, podcasts, books. Etc.)