Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sufficiently detailed spec is code

haskellforall.com

Post reply on HN
Print view View original

181–190 of 376 posts

Re: A sufficiently detailed spec is code

#181

Post by jumploops »

yes_man wrote: post #171
jumploops wrote: post #160

In my experience with “agentic engineering” the spec docs are often longer than the code itself. Natural language is imperfect, code is exact. The goal of specs is largely to maintain desired functionality over many iterations, something that pure code handles poorly. I’ve tried inline comments, tests, etc. but what works best is waterfall-style design docs that act as a second source of truth to the running code. Us…

> In my experience with “agentic engineering” the spec docs should be longer than the code itself. Natural language is imperfect, code is exact. The latter notion probably is true, but the prior isn’t necessarily true because you can map natural language to strict schemas. ”Implement an interface for TCP in ” is probably shorter than the actual implementation in code. And I understand my example is pedantic, but it e…

Thanks, I updated my comment to say “are often longer” because that’s what I see in practice.

To your point, there are some cases where a short description is sufficient and may have equal or less lines than code (frequently with helper functions utilizing well known packages).

In either case, we’re entering a new era of “compilers” (transpilers?), where they aren’t always correct/performant yet, but the change in tides is clear.

Re: A sufficiently detailed spec is code

#182

Post by jumploops »

amtamt wrote: post #161

> On two occasions I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. I guess many of us quality for british parliament.

Funnily enough, with the most recent models (having reduced sycophancy), putting in the wrong assumptions often still leads to the right output.

Re: A sufficiently detailed spec is code

#183

Post by prohobo »

lmm wrote: post #175
prohobo wrote: post #139

Why is everyone still talking about markdown files as the only form of spec? The argument is true for text-based specs, but that's not the only option. Stop being so text-file-brained? This article is really attacking vague prose that pushes ambiguity onto the agent - okay, fair enough. But that's a tooling problem. What if you could express structure and relationships at a higher level than text, or map domain conce…

> - 1970s: abstracted away the algorithm -> OOP, standard libraries, UML Which was mostly a failure, to the point that there is a major movement towards languages that "abstract away" (in this sense) less , e.g. Rust. Certainly if the creators of UML are saying that AI is great, that gives me more confidence than ever that it's bunk.

UML was for various reasons, but libraries? When's the last time you wrote a sorting algorithm? The entire software ecosystem runs on dependencies. That failed?

Rust uses crates to import those dependencies, which was one of its biggest innovations.

Re: A sufficiently detailed spec is code

#184

Post by internet_points »

half-kh-hacker wrote: post #73
abcde666777 wrote: post #16

Earlier quoted context omitted.

These algorithms don't have intelligence, they just regurgitate human intelligence that was in their training data. That also goes the other way - they can't produce intelligence that wasn't represented in their training input.

How does post-training via reinforcement learning factor in? Does every evaluated judgement count as 'the training data' ?

yes

Re: A sufficiently detailed spec is code

#185

Post by tibbe »

lmm wrote: post #173
bad_username wrote: post #145

> There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. Furthermore, LLMs are the ultimate detail fillers, because they are language interpolat…

> LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. They can generate boilerplate, sure. Or they can expand out a known/named algorithm implementation, like pulling in a library. But neither of those is generating detail that wasn't there in the original (at most it pulls in the detail from somewhere in the training set).

They do more than that. If you ask for ui with a button that button won't be upside down even if you didn't specify its orientation. Lots of the detail can be inferred from general human preferences, which are present in the LLMs' training data. This extends way beyond CS stuff like details of algorithm implementations.

Re: A sufficiently detailed spec is code

#186

Post by lukevp »

ozlikethewizard wrote: post #156
rytis wrote: post #128

Earlier quoted context omitted.

> if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.

The hard bit of programming has never been knowing the symbols to tell the computer what to do. It is more difficult to use a completely unknown language, sure, but the paradigms and problem solving approaches are identical and thats the actual work, not writing the correct words.

Saying that the paradigms of Python and Haskell are the same makes it sound like you don’t know either or both of those languages. They are not just syntactically different. The paradigms literally are different. Python is a high level duck typed oo scripting language and Haskell is a non-oo strongly typed functional programming language. They’re extremely far apart.

Re: A sufficiently detailed spec is code

#187

Post by TeeWEE »

There are two kid of specs, formal spec, and "Product requirements / technical designs"

Technical design docs are higher level than code, they are impricise but highlight an architectural direction. Blanks need to be filled in. AI Shines here.

Formal specs == code Some language shine in being very close to a formal spec. Yes functional languages.

But lets first discuss which kind of spec we talk about.

Re: A sufficiently detailed spec is code

#188

Post by cassianoleal »

rytis wrote: post #128
Gabriel439 wrote: post #109

Earlier quoted context omitted.

Author here: it's not even clear that agents can reliably permute their training data (I'm not saying that it's impossible or never happens but that it's not something we can take for granted as a reliable feature of agentic coding). As I mentioned in one of the footnotes in the post: > People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which…

> if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.

Your argument fails where it equates someone who only codes in one language to an LLM who is usually trained in many languages.

In my experience, a software engineer knows how to program and has experience in multiple languages. Someone with that level of experience tends to pick up new languages very quickly because they can apply the same abstract concepts and algorithms.

If an LLM that has a similar (or broader) data set of languages cannot generalise to an unknown language, then it stands to reason that it is indeed only capable of reproducing what’s already in its training data.

Re: A sufficiently detailed spec is code

#189

Post by hrmtst93837 »

charcircuit wrote: post #6

This articles ignores that AI agents have intelligence which means that they can figure out unspecified parts of the spec on their own. There is a lot of the design of software that I don't care about and I'm fine letting AI pick a reasonable approach.

[flagged]

Re: A sufficiently detailed spec is code

#190

Post by mike_hearn »

Someone wrote: post #157
bad_username wrote: post #145

> There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. Furthermore, LLMs are the ultimate detail fillers, because they are language interpolat…

> That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. LLMs can generate (relatively small amounts of) working code from relatively terse descriptions, but I don’t think they can do so _reliably_. They’re more reliable the shorter the code fragment and the more common the code, but they do break down for complex description…

Neither can humans, but the industry has decades of experience with how to instruct and guide human developer teams using specs.
Post reply on HN