Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why "just prompt better" doesn't work

bicameral-ai.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Why "just prompt better" doesn't work

#11

Post by clktmr »

siriusastrebe wrote: post #6

Can this be solved by a question answer session? You ask the coding assistant for a brand new feature. The coding assistant says, we have two or three or four different paths we could go about doing it. Maybe the coding assistant can recommend a specific one. Once you pick the option, the coding assistant can ask more specific questions. The database looks like this right now, should we modify this table which would…

You don't know what you want. That's why asking questions doesn't work. You think you know it, but only after you've spent some time iterating in the space of solutions, you'll see the path forward.

Re: Why "just prompt better" doesn't work

#12

Post by OrderlyTiamat »

clktmr wrote: post #11
siriusastrebe wrote: post #6

Can this be solved by a question answer session? You ask the coding assistant for a brand new feature. The coding assistant says, we have two or three or four different paths we could go about doing it. Maybe the coding assistant can recommend a specific one. Once you pick the option, the coding assistant can ask more specific questions. The database looks like this right now, should we modify this table which would…

You don't know what you want. That's why asking questions doesn't work. You think you know it, but only after you've spent some time iterating in the space of solutions, you'll see the path forward.

> You think you know it, but only after you've spent some time iterating in the space of solutions, you'll see the path forward.

I'd turn it around- this is the reason asking questions does work! When you don't know what you want, someone asking you for more specifics is sometimes very illuminating, whether that someone is real or not.

LLMs have played this role well for me in some situations, and atrociously in others.

Re: Why "just prompt better" doesn't work

#13

Post by bitwize »

noduerme wrote: post #3

I found this interesting: >> Small decisions have to be made by design/eng based on discovery of product constraints, but communicating this to stakeholders is hard and time consuming and often doesn’t work. This implies that a great deal of extraneous work and headaches result from the stakeholders not having a clear mental model of what they need software to do, versus what is either secondary or could be disposed…

Such an interlocutor was historically known as the systems analyst, but historically we've put programmers in that position, where they tend to do poorly.

https://www.linkedin.com/pulse/tail-wagging-dog-tim-bryce/

Systems analysis is about to make a roaring return, as the need for human programmers wanes thanks to LLMs generating all the code.

Re: Why "just prompt better" doesn't work

#15

Post by galaxyLogic »

OrderlyTiamat wrote: post #12
clktmr wrote: post #11

Earlier quoted context omitted.

You don't know what you want. That's why asking questions doesn't work. You think you know it, but only after you've spent some time iterating in the space of solutions, you'll see the path forward.

> You think you know it, but only after you've spent some time iterating in the space of solutions, you'll see the path forward. I'd turn it around- this is the reason asking questions does work! When you don't know what you want, someone asking you for more specifics is sometimes very illuminating, whether that someone is real or not. LLMs have played this role well for me in some situations, and atrociously in othe…

I think what's lacking in LLMs creating code is they can't "simulate" what a human user would experience while using the system. So they can't really evaluate alternative solutions tothe humna-app interaction.

We humans can imagine it in our mind because we have used the PC a lot. But it is still hard for use to anticipate how the actual system will feel for the end-users. Therefore we build a prototype and once we use the prototype we learn hey this can not possibly work productively. So we must try something else. The LLM does not try to use a virtual prototype and thne learn it is hard to use. Unlike Bill Clinton it doesn't feel our pain.

Re: Why "just prompt better" doesn't work

#16

Post by Foobar8568 »

jaggederest wrote: post #7
siriusastrebe wrote: post #6

Can this be solved by a question answer session? You ask the coding assistant for a brand new feature. The coding assistant says, we have two or three or four different paths we could go about doing it. Maybe the coding assistant can recommend a specific one. Once you pick the option, the coding assistant can ask more specific questions. The database looks like this right now, should we modify this table which would…

You're describing existing behavior of codex and claude at the moment, for what it's worth. They don't always catch every edge case (or even most) in depth or discuss things thoroughly, depending on the prompt, but if you say "ask questions and be sure to clarify any ambiguity or technical issues" they'll run right through many of the outstanding concerns.

And neither will really code to the "Spec". They will miss a few requirements even for a spec that is less long than your screen.

Codex seems to be more thorough for it, but needs a lot of baby sitting, Claude will be happy to tell you he is done while missing half of them but will implement through the stack.

Tests will be generally crap for both of them.

So while I am happy to have those, it doesn't replace development knowledge.

Claude will be happy to kill security features to make it works.

Re: Why "just prompt better" doesn't work

#17

Post by lemax »

I've tried all the Q&A skills, confidence meters and little hacks to get agents to clarify and propose better solutions. Clarification and planning has gotten a lot better using some skills (e.g. obra/superpowers), but counterproposals and negative feedback are rarely up to snuff with something a staff level colleague would come up with - this seems to be amplified when you already have an extensive PRD or plan together. If a plan is already fleshed out but is inefficient or contains some anti-patterns, I've had better results just throwing these out, taking what I've learned and summarizing tradeoffs in a brand new chat.

Once you have a comprehensive plan together, or a fairly full context window, agents have a lot of issues zooming out. This is particularly painful in some coding agents since they're loading your existing code into context and get weighted down heavily by what already exists (which makes them good at other tasks) vs. what may be significantly simpler and better for net-new stuff or areas of your codebase that are more nascent.

Re: Why "just prompt better" doesn't work

#18

Post by noduerme »

bitwize wrote: post #13
noduerme wrote: post #3

I found this interesting: >> Small decisions have to be made by design/eng based on discovery of product constraints, but communicating this to stakeholders is hard and time consuming and often doesn’t work. This implies that a great deal of extraneous work and headaches result from the stakeholders not having a clear mental model of what they need software to do, versus what is either secondary or could be disposed…

Such an interlocutor was historically known as the systems analyst, but historically we've put programmers in that position, where they tend to do poorly. https://www.linkedin.com/pulse/tail-wagging-dog-tim-bryce/ Systems analysis is about to make a roaring return, as the need for human programmers wanes thanks to LLMs generating all the code.

I could definitely see myself moving into that role officially in a lot of places where I already unofficially am responsible for it, but I honestly don't know how I'd approach herding coders who were relying on LLMs to the point of not fully understanding the code-level interreliance of the systems themselves. I'd still find myself doing my current job, but with more cognitive load reading Claude code and trying to understand how these things were stitched together.

There's something to be said for attention, and the window of attention provided by a human who wrote something rather than an LLM that has to guess the intention of it, every time it reboots.

Any coder who understands what they're building could theoretically be a systems analyst for the greater part of what their code is going to be embedded in. This is a weakly referenced argument, but, dropping the lower links in the chain and substituting them with LLMs is exactly where I think communication is bound to break down. Or: You can try to herd cats all day, but if you move up the chain and all you have below you is cats relying on LLMs, you've just shifted the same problem up to your level in the organization.

Re: Why "just prompt better" doesn't work

#19

Post by PeterStuer »

One could counter with "Why 'just meet better' doesn't work". ('stakeholders' aren't realy, they are not comfortable with the required level of detail, lack both in depth business domain, operational domain and technical domain knowledge, bikeshedding fiestas masking incompetence, ...)

So if the AI can surface misunderstandings through fast prototyping, this can cut trough lots of meeting BS.

In practice, the truth is somewhere in the middle as always.

Re: Why "just prompt better" doesn't work

#20

Post by Havoc »

For me as a hobby programmer that part - scoping, identifying pain points and planning - actually got massively better with coding agents.

Previously: send it and figure out issues in they fly

Now: write a half page description and ask the LLM to figure out what info is missing from my document to implement

That’s absolutely not the same as the team dynamics but in principle it seems to me that LLMs can do work that is directionally “here is a target state” and project manage towards that

Post reply on HN