[dead]
A sufficiently detailed spec is code
211–220 of 376 posts
Re: A sufficiently detailed spec is code
#212> There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. Furthermore, LLMs are the ultimate detail fillers, because they are language interpolat…
In get the sense that what you are responding to and even many comments to yours are expressing a kind of coping with the current dynamic, only exacerbated by the rather elitist and egoistic mentality that people in tech have had for a very long time now; i.e., they are falling…being pushed from Mt Olympus and there is A LOT of anxious rationalization going on. Not a mere 5 years ago even tech people were chortling d…
I use AI for my work every single day - and during some weekends too. Claude Code, with Opus. It is far from being able to reliably produce the code that we need for production. It produces code that looks ok most of the time, but I have seen it lose track of key details, misinterpret requisites and even ignore them sometimes - "on purpose", as in it writing something like "let's not do that requirement, it's not necessary".
This kind of thing happens at least once per day to me, maybe more.
I am not denying that it is useful, let me be clear. It is extremely convenient, especially for mechanical tasks. It has other advantages like quick exploration of other people's code, for example. If my employer didn't provide a corporate account for me, I would pay one from my own pocket.
That said, I agree with OP and the author that it is not reliable when producing code from specs. It does things right, I would say often. That might be good enough for some fields/people. It's good enough for me, too. I however review every line it produces, because I've seen it miss, often, as well.
Re: A sufficiently detailed spec is code
#213Earlier quoted context omitted.
Isn’t most standard software these days a permutation of things already done before?
But what's the point of re-building "standard software" if it is so standard that it already exists 100 times in the training data with slight variations?
Re: A sufficiently detailed spec is code
#214Writing code doesn't reliably work... However, as we converge toward a more collaborative development environment, code is more important than ever. Testing, sales, etc. more
important than ever. Security... more important them ever.
Or everything will converge toward a rust like inference optimized gibberish...
Re: A sufficiently detailed spec is code
#215Earlier quoted context omitted.
> LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. They can generate boilerplate, sure. Or they can expand out a known/named algorithm implementation, like pulling in a library. But neither of those is generating detail that wasn't there in the original (at most it pulls in the detail from somewhere in the training set).
They do more than that. If you ask for ui with a button that button won't be upside down even if you didn't specify its orientation. Lots of the detail can be inferred from general human preferences, which are present in the LLMs' training data. This extends way beyond CS stuff like details of algorithm implementations.
Isn't "not being upsidedown" just one of the default properties of a button in whatever GUI toolkit you are using? I'd be worried if an LLM _did_ start setting all the possible button properties.
Re: A sufficiently detailed spec is code
#216just like what djikstra said
Re: A sufficiently detailed spec is code
#217I've heard various suggestions of only committing spec.md or change requests in the git repo and using that as source of truth.
We have spent decades working on reproducible builds or deterministic compilation. To achieve this, all steps must be deterministic. LLMs are not deterministic. You need to commit source code.
Re: A sufficiently detailed spec is code
#218I think it's only a matter of time before people start trying to optimize model performance and token usage by creating their own more technical dialect of English (LLMSpeak or something). It will reduce both ambiguity and token usage by using a highly compressed vocabulary, where very precise concepts are packed into single words (monads are just monoids in the category of endofunctors, what's the problem?). Grammat…
Have you just reinvented programming languages and reinforced the author's point? Setting aside the problem of training, why bother prompting if you’re going to specify things so tightly that it resembles code?
[deleted]
Re: A sufficiently detailed spec is code
#219Trying to go the Spec -> LLM route is just a lost cause. And seems wasteful to me even if it worked.
LLM -> Spec is easier, especially with good tools that can communicate why the spec fails to validate/compile back to the LLM. Better languages that can codify things like what can actually be called at a certain part of the codebase, or describe highly detailed constraints on the data model, are just going to win out long term because models don't get tired trying to figure this stuff out and put the lego bricks in the right place to make the code work, and developers don't have to worry about UB or nasty bugs sneaking in at the edges.
With a good 'compilable spec' and documentation in/around it, the next LLM run can have an easier time figuring out what is going on.
Trying to create 'validated english' is just injecting a ton of complexity away from the area you are trying to get actual work done: the code that actually runs and does stuff.
Re: A sufficiently detailed spec is code
#220Earlier quoted context omitted.
In my experience, the further you move away from the user and toward the hardware and fundamental theoretical algorithms, the less true this becomes. This is very true for an email client, but very untrue for an innovative 3D rendering engine technology (just an example).
An email client is highly nontrivial, due to the complexities of the underlying standards, and how the real implementations you have to be compatible with don’t strictly follow them. Making an email client that doesn’t suck and is fully interoperable is quite an ambitious endeavor.
What complexities specifically? Implementing SMTP (from the client’s perspective) that other SMTP servers can understand is not very hard. I have done it. Does it follow every nuance of the standard? I don’t know, but it works for me. I haven’t implemented IMAP but I don’t see why it should be much harder. Is there a particular example you have in mind?