Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A sufficiently detailed spec is code

haskellforall.com

Post reply on HN
Print view View original

141–150 of 376 posts

Re: A sufficiently detailed spec is code

#141

Post by Archer6621 »

mfabbri77 wrote: post #117
layer8 wrote: post #92

Earlier quoted context omitted.

An email client is highly nontrivial, due to the complexities of the underlying standards, and how the real implementations you have to be compatible with don’t strictly follow them. Making an email client that doesn’t suck and is fully interoperable is quite an ambitious endeavor.

The point was to answer the question: "Can every piece of software be viewed as a permutation of software that has already been developed?" In my opinion, an email client is a more favorable example than a 3D engine. In fields where it is necessary to differentiate, improve, or innovate at the algorithmic level, where research and development play a fundamental role, it is not simply a matter of permuting software or…

Actually, in the specific case of a 3D program it's the current generation of LLM's complete lack of ability in spatial reasoning that prevents them from "understanding" what you want when you ask it to e.g. "make a camera that flies in the direction you are looking at".

It necessarily has to derive it from examples of cameras that fly forward that it knows about, without understanding the exact mathematical underpinnings that allow you to rotate a 3D perspective camera and move along its local coordinate system, let alone knowing how to verify whether its implementation functions as desired, often resulting in dysfunctional garbage. Even with a human in the loop that provides it with feedback and grounds it (I tried), it can't figure this out, and that's just a tiny example.

Math is precise, and an LLM's fuzzy approach is therefore a bad fit for it. It will need an obscene amount of examples to reliably "parrot" mathematical constructs.

Re: A sufficiently detailed spec is code

#142

Post by motoxpro »

I agree with this so much. And on top of this, I have the strong feeling that LLMs are BETTER at code than they are at english, so not only are you going from a lossy formate to a less-leossy format, you are specifying in a lossy, unskilled format.

Re: A sufficiently detailed spec is code

#143

Post by Animats »

hintymad wrote: post #28

> A sufficiently detailed spec is code This is exactly the argument in Brooks' No Silver Bullet. I still believe that it holds. However, my observation is that many people don't really need that level of details. When one prompts an AI to "write me a to-do list app", what they really mean is that "write me a to-do list app that is better that I have imagined so far", which does not really require detailed spec.

Not entirely.

For some problems, it is. Web front-end development, for example. If you specify what everything has to look like and what it does, that's close to code.

But there are classes of problems where the thing is easy to specify, but hard to do correctly, or fast, or reliably. Much low-level software is like that. Databases, file systems, even operating system kernels. Networking up to the transport layer. Garbage collection. Eventually-consistent systems. Parallel computation getting the same answer as serial computation. Those problems yield, with difficulty, to machine checked formalism.

In those areas, systems where AI components struggle to get code that will pass machine-checked proofs have potential.

Re: A sufficiently detailed spec is code

#144

Post by kstenerud »

trane_project wrote: post #98

I've been trying codex and claude code for the past month or so. Here's the workflow that I've ended up with for making significant changes. - Define the data structures in the code yourself. Add comments on what each struct/enum/field does. - Write the definitions of any classes/traits/functions/interfaces that you will add or change. Either leave the implementations empty or write them yourself if they end up being…

I've been working with coding LLMs for almost a year. Here's what I've found works best:

- Do a brainstorming session with the LLM about your idea. Flesh out the major points of the product, who the stakeholders are, what their motivations and goals are, what their pain points are. Research potential competitors. Find out what people are saying about them, especially the complaints.

- Build a high level design document with the LLM. Go through user workflows and scenarios to discern what kinds of data are needed, and at what scale.

- Do more market research to see how novel this approach is. Figure out what other approaches are used, and how successful they are. Get user pain points with each approach if you can. Then revisit your high level design.

- Start a technical design document with the LLM. Figure out who the actors of the system are, what their roles are in the system, and what kinds of data they'll need in order to do their job.

- Research the technologies that could help you build the system. Find out how popular they are, how reliable they are, how friendly they are (documentation, error messaging, support, etc), their long-term track record, etc. These go into a research document.

- Decide based on the research which technologies match your system best. Start a technical document with the LLM. Go through the user scenarios and see how the technologies fit.

- Decide on the data structures and flows through the system. Caching, load balancing, reliability, throughput requirements at the scale you plan to reach for your MVP and slightly beyond. Some UX requirements at this point are good as well.

- Start to flesh out your interfaces, both user and machine. Prototype some ideas and see how well they work.

- Circle back to research and design based on your findings. Iterate a few times and update the documents as you go using your LLM. Try to find ways to break it.

- Once you're happy with your design, build an architecture document that shows how the whole system will concretely fit together.

- Build an implementation plan. Run it through multiple critique rounds. Try to find ways to break it.

- Now you're at the threshold where changes get harder. Build the implementation piece by piece, checking to make sure they work as expected. This can be done quickly with multiple LLMs in parallel. Expect that the pieces won't fit and you'll need to rethink a lot of your assumptions. Code will change a LOT, so don't waste much time making it nice. You should have unit and integration tests and possibly e2e tests, which are cheap for the LLM to maintain, even if a lot of them suck.

- Depending on how the initial implementation went, decide whether to keep the codebase and refine it, or start the implementation over using the old codebase for lessons learned.

Basically, more of the same of what we've been doing for decades, just with faster tools.

Re: A sufficiently detailed spec is code

#145

Post by bad_username »

> There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail

That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions. Code is the detail being filled in. Furthermore, LLMs are the ultimate detail fillers, because they are language interpolation/extrapolation machines. And their popularity is precisely because they are usually very good at filling in details: LLMs use their vast knowledge to guess what detail to generate, so the result usually makes sense.

This doesn't detract much from the main point of the article though. Sometimes the interpolated detail is wrong (and indeterministic), so, if reliable result is to be achieved, important details have to be constrained, and for that they have to be specified. And whereas we have decades of tools and culture for coding, we largely don't have that for extremely detailed specs (except maybe at NASA or similar places). We could figure it out in the future, but we haven't yet.

Re: A sufficiently detailed spec is code

#146

Post by debugnik »

rytis wrote: post #128
Gabriel439 wrote: post #109

Earlier quoted context omitted.

Author here: it's not even clear that agents can reliably permute their training data (I'm not saying that it's impossible or never happens but that it's not something we can take for granted as a reliable feature of agentic coding). As I mentioned in one of the footnotes in the post: > People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which…

> if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.

But the model has seen pretty much all the public Haskell code around, and possibly been trained to write it in different settings.

Re: A sufficiently detailed spec is code

#147

Post by retsibsi »

ulrikrasmussen wrote: post #45

A corollary of this statement is that code without a spec is not code. No /s, I think that is true - code without a spec certainly does something, but it is, by the absence of a detailed spec, undefined behavior.

No way! Code is whatever defines the behavior of the program unambiguously, or as close to unambiguously as makes little difference. A sufficiently detailed spec does that, and hence it is effectively higher level code. Code (together with the language spec and/or compiler) always does that, regardless of how haphazardly written it is.

Re: A sufficiently detailed spec is code

#148

Post by Frieren »

rytis wrote: post #128
Gabriel439 wrote: post #109

Earlier quoted context omitted.

Author here: it's not even clear that agents can reliably permute their training data (I'm not saying that it's impossible or never happens but that it's not something we can take for granted as a reliable feature of agentic coding). As I mentioned in one of the footnotes in the post: > People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which…

> if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.

> doesn't that apply to flesh-and-bone developers?

No, it does not. If you have a developer that knows C++, Java, Haskell, etc. and you ask that developer to re-implement something from one language to another the result will be good. That is because a developer knows how to generalize from one language (e.g. C++) and then write something concrete in the other (e.g. Haskell).

Re: A sufficiently detailed spec is code

#150

Post by pixelbart »

lmm wrote: post #48
hintymad wrote: post #28

> A sufficiently detailed spec is code This is exactly the argument in Brooks' No Silver Bullet. I still believe that it holds. However, my observation is that many people don't really need that level of details. When one prompts an AI to "write me a to-do list app", what they really mean is that "write me a to-do list app that is better that I have imagined so far", which does not really require detailed spec.

> When one prompts an AI to "write me a to-do list app", what they really mean is that "write me a to-do list app that is better that I have imagined so far", which does not really require detailed spec. If someone was making a serious request for a to-do list app, they presumably want it to do something different from or better than the dozens of to-do list apps that are already out there. Which would require them t…

Would a musician refrain from writing a love song because there are already better love songs?
Post reply on HN