Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sufficiently detailed spec is code

haskellforall.com

Post reply on HN
Print view View original

331–340 of 376 posts

Re: A sufficiently detailed spec is code

#331

Post by derrak »

zombot wrote: post #263
randusername wrote: post #262

Safety-critical perspective: Specification means requirements. I like EARS [0] syntax for requirements. e.g. "while an error is present, the software shall ignore keypresses" Requirements are not code at all, they are expectations about what the code does; it is the contract about what developers will be held accountable to. Putting implementation details in requirements is a rookie mistake because it takes agency aw…

> for AI that leaves room for it to be more clever than we ourselves are. How probable is that given that LLMs output the average of their training data? Your own cleverness would have to be below average for this to hold.

> LLMs output the average of their training data

I think it follows that if you ask it to implement something, it will implement the average thing.

But then it also follows that if you give it the average thing and ask it to improve it by doing something clever, it may identify a clever improvement if such an improvement has ever been done in an analogous setting that exists in its training set.

> Your own cleverness would have to be below average

In this case the difference between myself and the LLM is that I’ve never seen the clever thing before (or I cannot recall it as effectively), but the LLM has and the LLM is specialized to recall it

Re: A sufficiently detailed spec is code

#332

Post by wizzwizz4 »

lukevp wrote: post #186
ozlikethewizard wrote: post #156

Earlier quoted context omitted.

The hard bit of programming has never been knowing the symbols to tell the computer what to do. It is more difficult to use a completely unknown language, sure, but the paradigms and problem solving approaches are identical and thats the actual work, not writing the correct words.

Saying that the paradigms of Python and Haskell are the same makes it sound like you don’t know either or both of those languages. They are not just syntactically different. The paradigms literally are different. Python is a high level duck typed oo scripting language and Haskell is a non-oo strongly typed functional programming language. They’re extremely far apart.

> The paradigms literally are different. […] They’re extremely far apart.

And yet, you can write pure-functional thunked streams in Python (and have the static type-checker enforce strong type checking), and high-level duck-typed OO with runtime polymorphism in Haskell.

The hardest part is getting a proper sum type going in Python, but ducktyping comes to the rescue. You can write `MyType = ConstructA | ConstructB | ConstructC` where each ConstructX type has a field like `discriminant: Literal[MyTypeDiscrim.A]`, but that's messy. (Technically, you can use the type itself as a discriminant, but that means you have to worry about subclasses; you can fix that by introducing an invariance constraint, or by forbidding subclasses….) It shouldn't be too hard to write a library to deal with this nicely, but I haven't found one. (https://pypi.org/project/pydantic-discriminated/ isn't quite the same thing, and its comparison table claims that everything else is worse.)

Re: A sufficiently detailed spec is code

#333

Post by derrak »

skydhash wrote: post #319
abainbridge wrote: post #309

Earlier quoted context omitted.

> Code is formal, and going from C to assembly is deterministic. OK, this is the main thing. Going from C to assembly is not deterministic in a sense because different compilers can produce different output. But the behaviour of the generated assembly is always the same. This isn't true of a spec.

> Going from C to assembly is not deterministic in a sense because different compilers can produce different output. That’s a weird definition of determinism.

Why is this a weird definition of determinism? Could you please define what you mean when you say deterministic?

A C program does not identify a single assembly program. It identifies a set of assembly programs. This fits the pretty standard definition of non-determinism.

A difference between natural language and C code is that natural language does not have a formal semantics. Having no formal semantics is a very different problem from having a semantics that admits a well-defined set of interpretations.

Re: A sufficiently detailed spec is code

#334

Post by isaachinman »

School-Cotton wrote: post #108
fmbb wrote: post #100

Earlier quoted context omitted.

I would be surprised if there are more working email clients out there than working 3D engines. The gaming market is huge, most people do not pay to use email, hobbyists love creating game engines.

Idk, a working basic email client is just not that hard to write though. SMTP and IMAP are simple protocols and the required graphical interface is a very straightforward combination of standard widgets.

I don't mean to be contrarian, but this is completely false.

IMAP _seems_ to be a straightforward (but nasty and stateful) protocol, until you find out that every major provider ignores RFCs and does things slightly differently.

It's a hellscape.

Re: A sufficiently detailed spec is code

#335

Post by scuff3d »

Seattle3503 wrote: post #138
scuff3d wrote: post #75

I recently left this comment on another thread. At the time I was focused on planning mode, but it applies here. Plan mode is a trap. It makes you feel like you're actually engineering a solution. Like you're making measured choices about implementation details. You're not, your just vibe coding with extra steps. I come from an electrical engineering background originally, and I've worked in aerospace most of my care…

I think of "plan" mode as a read-only mode where the LLM isn't chomping at the bit to start writing to files. Rather than being excitable and over-active, it is receptive and listening.

That's not really my point.

If you use it that way fine, but in general I'm talking about the idea that can you plan throughly enough in advance to get it to produce tens of thousands of lines of quality working code.

Like the author of the article points out, that takes so much effort by the time your done you may as well have just written the code.

Re: A sufficiently detailed spec is code

#336

Post by derrak »

riteshkew1001 wrote: post #317
angry_octet wrote: post #162

A spec is an envelope that contains all programs that comply. Creating this spec is often going to be harder than writing a single compliant program. Since every invocation of an LLM may create a different program, just like people, we will see that the spec will leave much room for good and bad implementations, and highlight the imprecision in the spec. Once we start using a particular implementation it often become…

[flagged]

> two programs can both satisfy the same spec and have completely different

The spec should express all relevant constraints. If your spec admits two things and only one is admissible in your mind, your spec is incomplete.

> has a massive envelope

The size of the envelope is less relevant than the expressivity of the language used to express subsets of that envelope. But almost always there is some logic which is more succinct for expressing the spec than the programming language used to express the implementation.

> your security posture is now a function of how exhaustive your spec is

The alternative is that your security posture is a function of unstated intentions living in somebody’s brain. This alternative seems strictly worse.

> You'd need to enumerate what NOT to do

This is equivalent to declaring what you must do and usually there is a succinct way to do this that does not involve listing every negative example

Re: A sufficiently detailed spec is code

#337

Post by FuckButtons »

mike_hearn wrote: post #190
Someone wrote: post #157

Earlier quoted context omitted.

> That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. LLMs can generate (relatively small amounts of) working code from relatively terse descriptions, but I don’t think they can do so _reliably_. They’re more reliable the shorter the code fragment and the more common the code, but they do break down for complex description…

Neither can humans, but the industry has decades of experience with how to instruct and guide human developer teams using specs.

You can guide humans, but ultimately the reason senior software developers have been payed large sums of money is that even with specs mostly we have found it works better to have someone with good judgement actually doing the work, otherwise we would have just been using specifications. The question remains open if llm’s can show good judgement, often my experience with claude is that it doesn’t if the problem domain is non-trivial but it’s possible that won’t always be true.

Re: A sufficiently detailed spec is code

#338

Post by derrak »

kikkupico wrote: post #54

Natural language is fluid and ambiguous while code is rigid and deterministic. Spec-driven development appears to be the best of both worlds. But really, it is the worst of both. LLMs are language models - their breakthrough capability is handling natural language. Code is meant to be unambiguous and deterministic. A spec is neither fluid nor deterministic.

> code is rigid and deterministic

I really wish people would start defining what they mean when they say code is deterministic. For instance, code is not deterministic in the sense that it admits a single interpretation when compiled. C code admits many assembly interpretations and the assembly you get will vary depending on your compiler and parameters.

When people say code is deterministic I think they mean it has a formal semantics. But these formal semantics may admit any one of many interpretations.

Re: A sufficiently detailed spec is code

#339

Post by skydhash »

derrak wrote: post #333
skydhash wrote: post #319

Earlier quoted context omitted.

> Going from C to assembly is not deterministic in a sense because different compilers can produce different output. That’s a weird definition of determinism.

Why is this a weird definition of determinism? Could you please define what you mean when you say deterministic? A C program does not identify a single assembly program. It identifies a set of assembly programs. This fits the pretty standard definition of non-determinism. A difference between natural language and C code is that natural language does not have a formal semantics. Having no formal semantics is a very di…

Determinism implies that the same input will result in the same output.

I agree with you that for a single C program there’s a set of assembly code that satisfies it. But by choosing a compiler, an architecture and a set of flags,… you will always get the same assembly code. If you decide to randomize them, then you can no longer guarantee a specific result, but you can still guarantee sets of result. Which is the definition of non-determinism.

Formalism is orthogonal as its about having well defined sets and transformation. LLMs are formal because it’s a finite set of weights and tokens ad the operations are well defined. But the prompt -> tokens -> tokens -> code transformation is non- deterministic in most tools (claude, chatgpt). And the relation between the input and the output os a mathematical one, not a semantic one.

Re: A sufficiently detailed spec is code

#340

Post by derrak »

skydhash wrote: post #261
fmap wrote: post #227

I agree with most of what the author is saying, but the slogan that "a sufficiently detailed spec is code" can be misunderstood as "a sufficiently detailed spec is a program". The statement is only true if you read "code" as "statement in a formal language". Here's a (sketch of a) specification for a compiler: > For every specification satisfied by the input program, the output program satisfies the same specificatio…

But that is not a specification for a compiler. That is just one of the requirement. It’s still missing all the error cases that can happen. Granted, most of those decisions will be taken at implementation time or during design discussions, but I believe if we noted them down, it would have been longer than the code. It’s just that people don’t bother and refer to the code for these kind of information.

> if we noted them down, it would have been longer than the code

If you are only noting down requirements of what the program is doing rather than how it should do that thing, I would expect that writing the requirements would necessarily be more succinct than writing the program.

It terrifies me every time I realize people are just jumping into writing code without specifying what that code is required to do. A day of hacking saves an hour of critical thought and planning.

Post reply on HN