Earlier quoted context omitted.
I’m convinced the magic bullet is deterministic checks. Linters, static analyzers, etc. Whatever you can do to create deterministic gates that the LLM simply must overcome to reach a “done” state, do it. Has been making a huge difference for my team, but sister teams are so invested in writing the perfect Make No Mistakes prompt that they just can’t see it. Basically I treat it like a junior dev. We don’t get junior…
Wouldn't have helped, sibling comment: https://news.ycombinator.com/item?id=48797883 Architectural decisions are not lintable.
I made a tool called ProjectLint to lint architectural decisions and anything that ESLint and ArchUnit and others like that wouldn't be a good fit for. It's not public yet, but is fairly rudimentary and based on Go + goja: https://github.com/dop251/goja
You write rules in ESLint, it makes sure that they're followed. If it's a file in your repo, it can obviously be checked. Now whether you can describe your architecture well enough to lint it (at least the stuff you care about enforcing), that's a different question.
In my case it's more like: "Oh, the AI messed up this pattern, but it will need to be followed N more times in the future, better write another projectlint rule." After a while, you can even copy them over between projects, as long as they're following the same conventions.
In the same spirit of https://www.archunit.org/