Agree, you need to your own code review, although as AI gets better, this problem will most likely be solved.
Be intentional about how AI changes your codebase
81–90 of 123 posts
Re: Be intentional about how AI changes your codebase
#82site is totally broken on mobile, just because cursor can vibe code a nice rolling scrolling shithole, chances are it will break on some platform/browser
Re: Be intentional about how AI changes your codebase
#83Earlier quoted context omitted.
Are there any good systems that somehow enforce consistency between documentation and code? Maybe the problem is fundamentally ill-posed.
I am not saying it doesn't matter because it does, but how much does it matter now since we can get documentation on the fly? I started working on something today I hadn't touched in a couple years. I asked for a summary of code structure, choices I made, why I made them, required inputs and expected outputs. Of course it wasn't perfect, but it was a very fast way to get back up to speed. Faster than picking through…
We cannot get full documentation on the fly, though. We can get "what this does" level of documentation for the system that AI is looking at. And if all you are doing is writing some code, maybe that is enough. But AI cannot offer the bigger picture of where it fits in the overall infrastructure, nor the business strategy. It cannot tell you why technical debt was chosen on some feature 5-10 years ago. And those types of documentation are far more important these days, as people write less of the code by hand.
This is the same discussion that goes round ad nauseum about comments. Nobody needs comments to tell us what the code does. We need comments to explain why choices were made.
Re: Be intentional about how AI changes your codebase
#84Earlier quoted context omitted.
There's nothing specific to AI about this. Humans make the same mistake. To solve this permanently, use a linter and apply a "ratchet" in CI so that the LLM cannot use ignore comments
Is there a Python linter that does this?
Not that I’m aware of (writing Python 10+ years). Suppose you could vibecode one yourself though.
Re: Be intentional about how AI changes your codebase
#85Earlier quoted context omitted.
> then it's all just feels Would that be so bad? "Readability" sure is subjective, so it seems "code quality" is. Ask 10 programmers what quality a snippet of code is, and you'll get 10 different answers.
And there is the problem. Then you start arguing about brace positions and function names and whether simple data classes should have docstrings on properties or not. All that time it's people arguing with people and wasting time on pure feels. People will get offended and angry and defensive, nothing good ever comes from it. But when you pick a style and enforce it with a tool like gofmt or black both locally and in…
> And there is the problem. Then you start arguing about brace positions and function names and whether simple data classes should have docstrings on properties or not.
Holy strawman Batman!
Have you ever given a code review? These are the lowest items on the totem pole of things usually considered critical for a code review.
Here’s an example code review from this week from me to a colleague, paraphrased:
“We should consider using fewer log statements and raising more exceptions in functions like this. This condition shouldn’t happen very often, and a failure of this service is more desirable than it silently chugging along but filling STDOUT with error messages.”
Re: Be intentional about how AI changes your codebase
#86Earlier quoted context omitted.
I don't think testing the product alone is good enough, because when you give it tests it has to pass it prioritizes passing them at the expense of everything else — including code quality. I've seen it pull in random variables, break semantic functions, etc.
Code quality can also be codified. If you can't express "code quality" deterministically, then it's all just feels. And if you can define "quality" in a way the agent can check against it, it will follow the instructions.
> Code quality can also be codified.
Do you think that no one has tried this over the past 80 years with human programmers, but now with LLMs we can suddenly manage to do it? Why do linters and formal verification and testing exist if we could’ve jus codified coding quality in the first place?
To me, this is like telling a carpenter that we can codify what makes a chair comfortable or not.
Re: Be intentional about how AI changes your codebase
#87Earlier quoted context omitted.
Simon Willison had this idea of "Documentation unit tests" in 2018: https://simonwillison.net/2018/Jul/28/documentation-unit-tes... It's not a massively complex AI monstrosity (it's from 2018 after all) or a perfect solution, but it's a good jumping off point. With a slight sprinkling of LLM this could be improved quite a bit. Not by having the agent write the documentation necessarily, but for checking the parity an…
interesting that they don’t mention doctest which has been a python built-in for quite a while. It allows you to write simple unit tests directly in your doc strings, by essentially copying the repl output so it doubles as an example. combined with something like sphinx that is almost exactly what you’re looking for. doctest kind of sucks for anything where you need to set up state, but if you’re writing functional c…
Doctest is writing unit tests in doctstrings.
That system is an unit test that checks that functions are documented in the documentation. Nothing to do with docstrings.
Re: Be intentional about how AI changes your codebase
#88Earlier quoted context omitted.
There's nothing specific to AI about this. Humans make the same mistake. To solve this permanently, use a linter and apply a "ratchet" in CI so that the LLM cannot use ignore comments
Is there a Python linter that does this?
I've been writing my own linter that's supposed to check projects regardless of the technology (e.g. something that focuses on architecture and conventions, alongside something like Oxlint/Oxfmt and Ruff and so on), with Go and goja: https://github.com/dop251/goja
Basically just a bunch of .js rules that are executed like:
projectlint run --rules-at ./projectlint-rules ./src
var WHAT_TO_DO = "If you absolutely need to disable an ESLint rule, you must follow the EXACT format:\n\n" +
"// prebuild-ignore-disallow-eslint-disable reason for disabling the rule below: [Your detailed justification here, at least 32 characters]\n" +
"// eslint-disable-next-line specific-rule-name\n\n" +
"Requirements:\n" +
"- Must be at least 32 characters long, to enforce someone doesn't leave just a ticket number\n" +
"- Must specify which rule(s) are being disabled (no blanket disables for ALL rules)\n" +
"- File-wide eslint-disable is not allowed\n\n" +
"This is done for long term maintainability of the codebase and to ensure conscious decisions about rule violations.";
And since the rules are there to prevent AI (or me) from doing stupid shit, they don't have to be super complex or perfect either, just usable for me. Furthermore, since it's Go, the executable ends up being a 10 MB tool I can put in CI container images, or on my local machine, and for example add pre-run checks for my app, so that when I try to launch it in a JetBrains IDE, it can also check for example whether my application configuration is actually correct for development.
Currently I have plenty in regards to disabling code checks, that reusable components should show up in a showcase page in the app, checking specific configuration for the back end for specific Git branches, how to use Pinia stores on the front end, that an API abstraction must be used instead of direct Axios or fetch, how Celery tasks must be handled, how the code has to be documented (and what code needs comments, what format) and so on.
Obviously the codebase is more or less slop so I don't have anything publish worthy atm, but anyone can make something like that in a weekend, to supplement already existing language-specific linters. Tbh ECMAScript is probably not the best choice, but hey, it's just code with some imports like:
// Standalone eslint-disable-next-line without prebuild-ignore
if (trimmed.indexOf("// eslint-disable-next-line") === 0) {
projectlint.error(file, "eslint-disable-next-line must be preceded by: " + IGNORE_MARKER, {
line: lineNum,
whatToDo: WHAT_TO_DO
});
continue;
}
Re: Be intentional about how AI changes your codebase
#89This could have been html instead of whatever awful moving pattern it is.
Wow. You weren’t joking.
Re: Be intentional about how AI changes your codebase
#90Earlier quoted context omitted.
And there is the problem. Then you start arguing about brace positions and function names and whether simple data classes should have docstrings on properties or not. All that time it's people arguing with people and wasting time on pure feels. People will get offended and angry and defensive, nothing good ever comes from it. But when you pick a style and enforce it with a tool like gofmt or black both locally and in…
> And there is the problem. Then you start arguing about brace positions and function names and whether simple data classes should have docstrings on properties or not. Holy strawman Batman! Have you ever given a code review? These are the lowest items on the totem pole of things usually considered critical for a code review. Here’s an example code review from this week from me to a colleague, paraphrased: “We should…
So you're fine with people using, for example, different brace styles at random? Or one person uses var everywhere, other uses definite types. One adds standard docstrings on every function and property, one never comments a single line of code.
Don't you have "format on save" enabled in your editor? When you open a file, change two lines and save -> boom 500 changed lines because the previous programmer had different formatting rules than you. Whoops.
This is why the low totem pole stuff needs to be enforced automatically so that actual humans can focus on the higher stuff that's about feels and intuition - things that are highly context dependent and can't be codified into rules.