Only frauds/vibecoders/non-programmers claim coding is "solved". This is nonsense.
Measuring the sloppiness of code
211–220 of 247 posts
Re: Measuring the sloppiness of code
#212Goodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
> could we throw the definition into context and tell the LLM to avoid it?
Every IMPORTANT RULE WHICH MUST BE FOLLOWED STRICTLY to not to do something from my AGENTS.md and skills are violated without any hesitation from LLM. Glorified autocomplete doesn't work this way. Positive rules are more stable on this regard.
Re: Measuring the sloppiness of code
#213I hopped into this hoping it would land where it did! I recently built in basic observations about cyclomatic complexity, churn, and authorship into an 'analysis' view of codebases in ouijit[1].
Some benefits can be reaped immediately (cyclomatic complexity), but others emerge over time (churn). A good example of this is say something like a 500 line file that has experienced 2500 lines of churn, and if that rate of churn is trending down or up. Surprisingly useful for understanding you've got a hotspot with an opportunity to pay down debt by spending more time on API design, or just breaking out whatever subset of the code is experiencing thrash.
The funny thing about complexity is that assuming you're lintting/formatting well, you can do a poor mans check by just looking at something like average indentation per line, deepest line, etc.
[1]: https://ouijit.com
Re: Measuring the sloppiness of code
#214I hope this is not too provocative, but where I work the legacy code is such a spaghetti mess and so actively inefficient (queries in loops of loops and plenty of things like this) that just prompt nowaday llm to "write extensive and menignful tests for this module, then rewrite the module clean mvc small functions , small loc per files, well organized files then make pass tests again" often leads to insanely better ai "sloppy" resulting code, plus free test coverage where nothing existed, users instant satisfaction (software become suddenly responsive in less than a second).
From this point it take not much more time to add proper ci/cd suite, rework database models and migrate things with now some comfidence.
It works very well on small isolated projects and new builds where resulting tools are produced insanely faster than before and with not comparable average code quality (ai slop is 100x better here) from customer usual habits and expectations.
It's because this approach raise significantly the quality level so quickly and so cheap that I advocate for it.
It however does not work the same way for large legacy projects to revive. For larger projects the reciepe is almost the same but requires significantly more manual testing on each pass and carefull regression check. But it does work.
Customer is happy, they take back "more" control over their legacy horrors at the cost of some extra loc but properly working code. There is no point to fight against "a bit too much code" in my case
I don't want to speak for those that work in elite very sensitive human responsibilities software domains, but in a lot of place this is a revolution in my opinion.
Re: Measuring the sloppiness of code
#215The conclusion is not surprising really, because fundamentally how do you even quantify sloppiness, a famously broad and subjective characterization?
I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality.
People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality."
An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures.
But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs!
Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow.
So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
Re: Measuring the sloppiness of code
#216Earlier quoted context omitted.
I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself? My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.
A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middl…
Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!)
As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
Re: Measuring the sloppiness of code
#217Coding is solved, perhaps, with unlimited token spend on a frontier model. It remains to be seen if it that is prohibitively expensive forever. At my company, we token maxed while the getting was good. But when we had to switch to Anthropic's enterprise plan, and start paying per token, the shit really hit the fan. Now we're retreating back to sane cost levels and finding that - guess what? - people power might just…
How did the developers react when they had to go back to writing their own code instead of playing Nintendo Switch between prompting sessions?
> had to go back to writing their own code
I hate it. My fingers are cramping and my head is hurting.
Re: Measuring the sloppiness of code
#218Plopping in my email to the author below in case anyone else is interested in this kind of thing: > There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been wo…
That's a very interesting idea for a benchmark!
Thank you! My favorite bit (theoretically) would be the forcing function to produce useful open source goods as a result of running these benchmarks and quiet down all the hype about llms. We get it, they’re very good at making 3d models do dumb shit that doesn’t matter
Re: Measuring the sloppiness of code
#219This is once again RLHF loops. the AI labs are and have been 100% focused on correctness because it is easy to setup and validate. Adding one more function that almost does the same thing as another will not break anything. I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to…
Maintainability is hard because it requires much more planning than correctness. You need to think about how you can construct code which gives you the most amount of reusability while not compromising on readability and also thinking about future paths and how they are affected. It seems like solving the problem of constructing maintainable code is as hard as solving long scale planning in LLMs
Re: Measuring the sloppiness of code
#220Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anym…
In theory - if an LLM could handle infinite complexity, I still think that the business issues + decisions end up getting in the way somewhere. AI: "You asked to add feature X. Here are 25 questions that impact feature Z, B, and C in your gigantic codebase" Developer: I can answer 8 of these questions... Guess I need to go figure out the rest of them. Writing the code + building the functionality has always been the…
Yeah, exactly my experience. Especially ever since Fable came out, I felt paralysed, many times, about how many decisions I have to hand it. Which is 10x harder when you handed it the reins to build the PR and you practically only knew the compressed initial problem statement. Then it stands up a draft PR and says: "Decisions you owe me: [insert bulleted list with 8 items each a paragraph long]" and you are like: "...oh shit".
It becomes a very heavy and difficult exercise of it walking you through the implementation and the judgment calls it had to make along the way. VERY exhausting.
There's a silver lining though: you do get to gradually clarify a proto project spec and various requirements, but boy does it take time and energy to re-contextualise when the bot tells you that you should make decisions.