71–74 of 74 posts
What I'd like to see is what it is, how and why it works, ideally backed by some benchmarks. Otherwise it looks like just another pile of skills with unknown outcomes.
Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code.
Basically the old theory is true - the code IS the specification.
What I did find very useful and have retained is the process flow. Create a proposal, review the proposal, implement, review the code. Also useful was building and maintaining ADRs and invariant logs for where a unit test cannot be made to verify behaviour. The process and the ADRs, unit test, invariant log all help the software stay coherent as the LLM churns on it over many unconnected contexts.
Can someone tell me how this stuff is any different from the 90's let's use UML for code gen and how it's not going to fail in the same way? In my experience, spec drift is the main reason why none of these tools work. Maybe they work for one shot greenfield feature generation but in a large, multi developer long lived code bases, specs rot and end up being more pain than they are worth.
1. Lack of closed loop between the "spec" and working code (your spec rot point). The Rational Unified Process(RUP) was a grossly inefficient, heavily manual undertaking. Mapping between artefacts - e.g. "Platform Independent Models" and "Platform Specific Models" was a manual, largely heuristic based approach. As a consequence the models were not generally kept up to date as the project evolved.
2. User experience mismatch. Developers were asked to create diagrams instead of writing code. Tool usability was poor ("write code with a mouse") and the artefacts didn't fit well with necessary tools like diffing and source code control (try diffing an xml file textually).
Coding agents have some potential for alleviating (1) in that they can read the result code and, at least to some extent, ensure spec and code are in sync.
(2) is more open. Some users - those proportionally more interested in solving the problem than designing/writing code - are more comfortable with natural-language-based specs and exploration. Those more experienced/comfortable with code will likely see those specs more akin to UML diagrams: a distraction from the real thing.
I just have it write a checklist file in /tmp (or a todo folder if I want to keep it), and check off items as it finishes them. Seems to work fine. Is this really needed?
/tmp? What happens when you reboot?