71–80 of 84 posts
What I'd like to see is what it is, how and why it works, ideally backed by some benchmarks. Otherwise it looks like just another pile of skills with unknown outcomes.
Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code.
Basically the old theory is true - the code IS the specification.
What I did find very useful and have retained is the process flow. Create a proposal, review the proposal, implement, review the code. Also useful was building and maintaining ADRs and invariant logs for where a unit test cannot be made to verify behaviour. The process and the ADRs, unit test, invariant log all help the software stay coherent as the LLM churns on it over many unconnected contexts.
Can someone tell me how this stuff is any different from the 90's let's use UML for code gen and how it's not going to fail in the same way? In my experience, spec drift is the main reason why none of these tools work. Maybe they work for one shot greenfield feature generation but in a large, multi developer long lived code bases, specs rot and end up being more pain than they are worth.
1. Lack of closed loop between the "spec" and working code (your spec rot point). The Rational Unified Process(RUP) was a grossly inefficient, heavily manual undertaking. Mapping between artefacts - e.g. "Platform Independent Models" and "Platform Specific Models" was a manual, largely heuristic based approach. As a consequence the models were not generally kept up to date as the project evolved.
2. User experience mismatch. Developers were asked to create diagrams instead of writing code. Tool usability was poor ("write code with a mouse") and the artefacts didn't fit well with necessary tools like diffing and source code control (try diffing an xml file textually).
Coding agents have some potential for alleviating (1) in that they can read the result code and, at least to some extent, ensure spec and code are in sync.
(2) is more open. Some users - those proportionally more interested in solving the problem than designing/writing code - are more comfortable with natural-language-based specs and exploration. Those more experienced/comfortable with code will likely see those specs more akin to UML diagrams: a distraction from the real thing.
I just have it write a checklist file in /tmp (or a todo folder if I want to keep it), and check off items as it finishes them. Seems to work fine. Is this really needed?
/tmp? What happens when you reboot?
I just have it write a checklist file in /tmp (or a todo folder if I want to keep it), and check off items as it finishes them. Seems to work fine. Is this really needed?
If you're writing design documentation to /tmp you're not the target audience for this kind of thing
Most of the time all you need is a throwaway md file to compensate for context loss and so you can loop on a spec across compaction. Once you have the spec implemented, there's usually no reason to keep it around. The code becomes the spec.
For hard constraints you have AGENTS.md, or put it in another md file. Why do I need additional software for this?
I've used OpenSpec extensively on two large pieces of software which were worked solo over 6-9 months. Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code. Basica…
Been using it for a bit. I think its a matter of time until we dont look at code anymore and only look at specs, plans, design docs. I didnt like the doc organization (or lack of it) using a standard reader or editor, so I built one specifically for OpenSpec. Hope you find it useful https://specks.nicotejera.com
I've used OpenSpec extensively on two large pieces of software which were worked solo over 6-9 months. Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code. Basica…
my org at work adopted openspec, and i strongly dislike it. every change, medium or larger, turns into a large set of multiple markdown documents, each that need review. and they are never handwritten - always slop, filled with the all the tells of ai writing, which i personally find grating. i find a small, human written spec to be much more effective than these large spec documents. the idea is that you iterate wit…
I've used OpenSpec extensively on two large pieces of software which were worked solo over 6-9 months. Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code. Basica…
The spec is whatever I write by hand. The code is what the LLM writes for me. The spec could be anything depending on how much detail you want.
The problem with the "code IS the spec" in the age of LLMs is that they will change stuff without telling you while hitting their immediate goal. Six months ago, I used to review every single change. Now I get the LLM to audit the code to compare against the spec. Any divergence means one of two things:
- either I have to update the spec, or
- the LLM has to update the code.