201–210 of 246 posts
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff... It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and…
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team. If humans increasingly are kept out of coding, then who holds the mental model? If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM spea…
Who says anyone needs to hold a mental model? Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase. Before LLMs we couldn’t have built tools that would deliver such views, now we can.
The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide.
We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects.
Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anym…
Something as simple as output length is a hard linear floor for productivity, even putting aside the obvious context problems that you're intuiting, and it's far from being the biggest cost that arises from steering skill. Learning to make a smaller, faster model do the same work with less tokens is a technical domain that a lot of people don't seem capable of learning. I'm not just talking about "context engineering", but learning how to fine tune, post-train, create better harnesses, design inference setups, etc. If we're both using AI, but I'm beating you to market every single time and with a better product, what is your AI usage actually buying you? Yes, competency and skill is this meaningful right now, and it's highly technical. Not the least of which because you know how to describe the problem in way that gives it a smaller solution and requires less iteration.
Most of the labor who understand the technology enough to do those things lives at the companies selling you these services, but you can absolutely learn to do these things yourself right now. It's actually really fun! A hell of a lot more fun than fucking prompting that's for sure.
Where we're at, I would equate it to the early mainframe era where the programmers came with the computer. I'm placing calls that we follow a similar track and the two will end up decoupling, that "model engineers" are going to move in-house. OpenAI will have a ring to it like IBM does today.
Goodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
Code that is overly and unnecessarily difficult and cumbersome for humans to work on.
But given the way humans work on code nowadays doesn't involve actually writing it themselves, there's no such thing as sloppy code anymore. Hooray!
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team. If humans increasingly are kept out of coding, then who holds the mental model? If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM spea…
Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operating with such a model.
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facil…
I does the thing, adds some code, adds test that cover that code, and responds with "Done."
At a fully agentic "No looking at code" company, we now don't have a great understanding of that code, we don't know whether it's a good implementation or not, whether it fits our current patterns, is maintainable, or composable.
We're entirely in the dark, at the behest of an agent at this point - if we want to maintain velocity. (And I would love to know whether that velocity is real or just perceived). It feels like the code part is faster? But also feels like I have to spend a lot more time up front working through a problem to understand it. In the past, I gained that understanding WHILE I was working through the code. Who knows.
Really glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start! My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a…