Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sufficiently detailed spec is code

haskellforall.com

Post reply on HN
Print view View original

361–370 of 376 posts

Re: A sufficiently detailed spec is code

#361

Post by jongjong »

serallak wrote: post #282
jongjong wrote: post #229

This is relatable. I did a side project with a non-technical co-founder a year ago and every time he told me what he wanted, I made a list of like 9 or 10 logical contradictions in his requirements and I had to walk him through what he said with drawings of the UI so that he would understand. Some stuff he wanted me to do sounded good in his head but once you walk through the implementation details, the solution is e…

He wanted seven perpendicular lines ?

Haha. I was like "Glad you've got this 4D chess going on in your head but the user can only see in 3D, buddy."

Re: A sufficiently detailed spec is code

#362

Post by mike_hearn »

reedlaw wrote: post #329
mike_hearn wrote: post #192

Earlier quoted context omitted.

Your argument is far too dependent on observations made about the model's ability with Haskell, which is irrelevant. The concepts in Haskell are totally different to almost any other language - you can't easily "generalize" from an imperative strict language like basically everything people really use to a lazy pure FP language that uses monads for IO like Haskell. The underlying concepts themselves are different and…

Why is Haskell irrelevant to the argument that LLMs can't reliably permute programming knowledge from one language to another? In fact, the purity of the language and dearth of training data seems like the perfect test case to see whether concepts found in more mainstream languages are actually understood.

Because human programmers routinely fail to do that too. Haskell is an obscure language that came out of academic research. Several of the core semantics (like lazyness by default) didn't get adopted anywhere else and are only found in Haskell.

Re: A sufficiently detailed spec is code

#364

Post by reedlaw »

mike_hearn wrote: post #362
reedlaw wrote: post #329

Earlier quoted context omitted.

Why is Haskell irrelevant to the argument that LLMs can't reliably permute programming knowledge from one language to another? In fact, the purity of the language and dearth of training data seems like the perfect test case to see whether concepts found in more mainstream languages are actually understood.

Because human programmers routinely fail to do that too. Haskell is an obscure language that came out of academic research. Several of the core semantics (like lazyness by default) didn't get adopted anywhere else and are only found in Haskell.

Then I would say this is another proof that LLMs lack intellect or ability to reason about universals. See https://michaelmangialardi.substack.com/i/186405810/test-4-p...

Re: A sufficiently detailed spec is code

#365

Post by tinodb »

strix_varius wrote: post #352
bad_username wrote: post #145

> There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. Furthermore, LLMs are the ultimate detail fillers, because they are language interpolat…

> the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions > Sometimes the interpolated detail is wrong (and indeterministic) ... You consider incorrect, non deterministic results to be "reliable"?

Do you consider the implementation of such specs by another human to (always) be correct and deterministic?

Heck, if I reimplement something I worked on a month ago it’s probably not going to be the exact same. Being non deterministic needn’t to be a problem, as long as it falls within certain boundaries and produces working results.

Re: A sufficiently detailed spec is code

#366

Post by itsfine2 »

bigfishrunning wrote: post #310
jappgar wrote: post #246

Earlier quoted context omitted.

As you can see by downvotes and comments, they still don't get it. LLMs make developers more efficient. That much is obvious to anyone who isn't blinded by fear. But people will respond "but you still need developers!" True. You don't need nearly as many, though. In fact, with an LLM in their hands, the poor performers are more of a liability than ever. They'll be let go first. But even the "smart" developers will be…

> LLMs make developers more efficient. They do not. I review a ton of code, and while the quantity is going up, the quality of that code is getting worse. LLMs only make developers more efficient if they skip the due-diligence required to verify its output; they all say they don't, and almost all of them do.

Probably it's not a perston you're answering to, so there no point to try to have a reasonable conversation.

Re: A sufficiently detailed spec is code

#367

Post by derrak »

kikkupico wrote: post #349
derrak wrote: post #338

Earlier quoted context omitted.

> code is rigid and deterministic I really wish people would start defining what they mean when they say code is deterministic. For instance, code is not deterministic in the sense that it admits a single interpretation when compiled. C code admits many assembly interpretations and the assembly you get will vary depending on your compiler and parameters. When people say code is deterministic I think they mean it has…

> C code admits many assembly interpretations and the assembly you get will vary depending on your compiler and parameters. There's your own example defining how code is deterministic. Despite being compiled to different assembly interpretations, the code does the same thing.

All compilations do what I intended to do when I wrote the C code. But each compilation does the intended thing in a different way.

The point I’m making is that determinism isn’t the key difference between compilers and LLMs. The key difference is that LLMs do not give semantic guarantees. I don’t care if they give the same output on the same input, just as I don’t care very much which C compiler I use.

Re: A sufficiently detailed spec is code

#368

Post by thienannguyencv »

interstice wrote: post #238
pablobaz wrote: post #231

Earlier quoted context omitted.

which bits of this do you think llm based agents can't do?

Not get stuck on an incorrect train of thought, not ignore core instructions in favour of training data like breaking naming conventions across sessions or long contexts, not confidently state "I completely understand the problem and this will definitely work this time" for the 5th time without actually checking. I could go on.

[dead]

Re: A sufficiently detailed spec is code

#369

Post by thienannguyencv »

abcde666777 wrote: post #16
charcircuit wrote: post #6

This articles ignores that AI agents have intelligence which means that they can figure out unspecified parts of the spec on their own. There is a lot of the design of software that I don't care about and I'm fine letting AI pick a reasonable approach.

These algorithms don't have intelligence, they just regurgitate human intelligence that was in their training data. That also goes the other way - they can't produce intelligence that wasn't represented in their training input.

"regurgitate human intelligence from training data" - exactly. And the tricky part is when your actual code contradicts that training data.

Model saw thousands of examples of "how to implement X". When your codebase does X differently, training data wins. You can see it happen: point out the conflict, and a model that's reasoning would shift gears - ask questions, acknowledge tension. But model in retrieval mode just reiterates. Same confidence, same explanation, maybe rephrased.

That's why "I completely understand this time" keeps happening in AI responses. From model's view, nothing to check - the pattern it retrieved already "makes sense."

In short, if you're not doing something completely new, something that AI will almost certainly do better than you, then you're safe. Otherwise, you'll have to put in a considerable amount of effort to get the AI to cooperate, or you'll have to do the most difficult tasks yourself, simply because the limitations described above haven't been addressed.

Re: A sufficiently detailed spec is code

#370

Post by kikkupico »

derrak wrote: post #367
kikkupico wrote: post #349

Earlier quoted context omitted.

> C code admits many assembly interpretations and the assembly you get will vary depending on your compiler and parameters. There's your own example defining how code is deterministic. Despite being compiled to different assembly interpretations, the code does the same thing.

All compilations do what I intended to do when I wrote the C code. But each compilation does the intended thing in a different way. The point I’m making is that determinism isn’t the key difference between compilers and LLMs. The key difference is that LLMs do not give semantic guarantees. I don’t care if they give the same output on the same input, just as I don’t care very much which C compiler I use.

> The point I’m making is that determinism isn’t the key difference between compilers and LLMs

I think you misunderstood. Determinism is the the difference between natural language and code. It's not about LLMs vs compilers.

Post reply on HN