Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An example of LLM prompting for programming

martinfowler.com

Post reply on HN
Print view View original

171–180 of 297 posts

Re: An example of LLM prompting for programming

#171

Post by spaceman_2020 »

gdubs wrote: post #154

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwago…

People also forget that the model is trained on older data. At first, it will default to referencing out of date frameworks and solutions, but if you tell it that its code isn't working, it will usually correct itself.

Re: An example of LLM prompting for programming

#172

Post by tjr »

agentultra wrote: post #133
nextworddev wrote: post #130

Earlier quoted context omitted.

Arguably reading code can’t lead to definitive conclusions about its bug-free-ness

Precisely! And neither can generating a handful of unit tests. As EWD would say, they only prove the existence of one error. Not that there are no errors. If we want more programs that are correct with respect to their specifications we need to write better, precise specifications… not wave our hands around. However for a lot of line-of-business tasks we’re generally fine with ambiguous, informal specifications. We’r…

This is true in aerospace software. Lots of process, lots of specification, lots of verification. I wouldn't want to say that GPT-seque tools would be useless here, but I really don't see them offering the same kind of magic leverage that they might offer on some other projects.

And vice-versa! Most software projects do not benefit from the rigor used in aerospace, because it's just not needed, and would be a waste of time.

I am definitely seeing ways that GPT tools could speed up some aerospace work, but we need to be really really sure that things are being done correctly... not just mostly correct, or seemingly correct.

Re: An example of LLM prompting for programming

#173

Post by drowsspa »

rootusrootus wrote: post #155
gdubs wrote: post #154

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwago…

I agree it's a good idea to take a moderate approach. The hype that LLMs are going to replace SWEs is clearly just that, hype, if you've done any real work trying to get GPT4 to give you the code you want. But it's also clearly a very useful tool. I think it'll absolutely destroy Stack Overflow.

Yet, the whole movement of getting blue collar workers to code seems to have lost its steam.

Re: An example of LLM prompting for programming

#174

Post by scarface74 »

mjr00 wrote: post #56
yodsanklai wrote: post #50

Earlier quoted context omitted.

> Copilot is much better for someone actually writing code I haven't used copilot yet, but I'm using occasionally chatgpt with prompts such as "write a bash/python script take takes these parameters and perform this tasks". Then I iterate if needed, and usually, i can get what i want faster than without using chatgpt. It's not a game changer, but it's a performance boost. How natural language is a distraction here? a…

Try not using natural language and just type what you'd type into Google. You'll get the same results and realize that all of the natural language fluff is totally unnecessary. I just typed in "bash script recursive chmod 777 all files" (as a dumb toy example) and got a resulting script back. It was surrounded by two natural language GPT comments: > It's generally not recommended to give all files and directories the…

> Try not using natural language and just type what you'd type into Google. You'll get the same results and realize that all of the natural language fluff is totally unnecessary.

I can get similar results with Google sometimes and I can put together what I learned from different places.

But I can get scripts that meet my exact requirements with ChatGPT. Most of my ChatGPT related code is scripting AWS related code and CloudFormation templates.

I’ve asked it to translate AWS related Python code to Node for a different projects and a bash shell script. It’s well trained on AWS related code.

I don’t know PowerShell from a hole in the wall. But I needed to write PS scripts and it did it. I’ve also used it to convert CloudFormation to Terraform

Re: An example of LLM prompting for programming

#176

Post by davidthewatson »

zackmorris wrote: post #134

This is an amazing demonstration, but I'm worried that when this goes mainstream, we'll inherit a ton of baggage from today's programming. Specifically: * The tests are written in BDD style "it('should xyz')", which programmers do in code like this for convenience. But if we're automating their creation, then actual human-readable Cucumber clauses would be more useful. Maybe the tests can be transpiled. This isn't th…

This is deeply resonant with me for the following reasons:

1) Age

2) BDD-style or what I call a madlib proxy for playing cucumber on TV. Not a fan having used it in an RoR context I can only call hipster-engineering, not what DHH described.

3) I just had the discussion on redux vs. datomic vs. riak with friends yesterday.

4) Ditto the conversation on MVVM and the implied constraint complexity of putting nodejs and chromium in the same deployment package and calling it electron while carrying on how simple it is relative to... a world where everything is actually native all the way down?

5) Me too on the CASE era.

6) Cue Donald Knuth on literate programming. One thing that cucumber is not, but I think taking another iteration at literate programming in light of GPT or LLMs is a good idea since Knuth is never wrong just 50 years ahead of time, but we needed a collaboration of human-computer agents that is patterned on a sensemaking protocol that can resolve subjective truth by consensus of man and machine. How else could you possibly resolve the fact that the SOTA lies to me on a daily basis while defending itself and its lack of veracity with force in what can only be seen as emulating the culture of one's parents.

7) Yes, AI should help on the iterations. Those short design sketch-to-demo we used to do at the design studio with sketch on Monday and demo on Friday should be much easier today to go from breakfast sketch to dinner demo, but I don't think they are. The tooling is radically better but that better has come at the cost of complexity and going sideways, neither of which are being fully felt and accounted for reflectively, i.e. they're not how you get to typing less and having the tools do the work because when they break, the debugging is mind-crushing.

8) I think the thing that's missing in the trivial part is that it's not actually trivial, but particularly because the software is the message and that insight stems from the fact that software has emergent properties such as extensibility, composability, and a resultant rate of change that make it very difficult to compare from decade to decade because software's fundamental disequilibrium stems from the fact that the full stack is in constant flux from a mad hatter's pop culture where we never sing the same song twice. There's value in theme and variations if it can be modeled as improvisational human-computer design pairing rather than yet another orchestration. Joe Beda was as right about improvisation as Knuth is about the art of computer programming.

9) I guess the t-shirt is: I'm not waiting...

10) In the immortal words of Raymond Loewy: Never leave well enough alone.

If there's a set of artifacts in software that achieve what I hope for with AI, it's somewhere between Bret Victor and https://iolanguage.org/

Re: An example of LLM prompting for programming

#177

Post by lcnPylGDnU4H9OF »

svachalek wrote: post #164
lcnPylGDnU4H9OF wrote: post #161

Earlier quoted context omitted.

> destroy Stack Overflow It'll be interesting to see how future training data is sourced.

You simply need the system to train itself on its own interactions, like how search engines improve results by counting clicks.

I'm not wondering about how the system will determine what's most helpful but instead determining what's even "correct". A model will learn what's "correct" from Stack Overflow by finding accepted or highly-voted answers but when it can't find such content anymore (in this case because Stack Overflow is hypothetically gone) then what would even exist to generate these discussions to be used as training data?

Github, per the sibling comment, is a good example because projects will have issues (tied to the individual repository of source code to be seen as a working implementation of the idea) which will be where such discussions happen.

Re: An example of LLM prompting for programming

#178

Post by afro88 »

The article shows everything that works for this approach. But it's a bit disingenuous. At the end:

> Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan.

No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of this quality code and conversation - ChatGPT). The LLM will start hallucinating the task list, the names of functions it wrote earlier etc. and the responses would get less and less useful with more and more "this doesn't work, can you fix X".

So anyone following this approach will hit a footgun after task 1.

For anyone that really wants to follow this approach, the next step is to start a new chat and copy/paste the inital requirement prompt, put the task list in there, any relevant code, adjust the instruction (ie "help me with task 2") and go from there.

It is of limited utility though. By step 3 (or even 2) you end up with so much code that you're at the token limit anyway and it can't write code that fits together.

Where I've found ChatGPT 4 useful is getting me going on something, providing boilerplate, and unblocking me.

If you don't know how to approach a problem like the "awareness layer" (like I didn't before reading the post), you can get a great breakdown and starting point from ChatGPT. Similarly, if you're not sure how to approach that view model, or write tests etc. And if you want a first draft of code or tests.

All that said, I'm looking forward to much larger and affordable token limits in future.

Re: An example of LLM prompting for programming

#179

Post by baq »

wpietri wrote: post #152
themodelplumber wrote: post #8

If somebody thinks an LLM is coming for everybody's coding job, I'd say this article is a great counterpoint just for existing. You could tell someone from decades ago that we now use a very high level language for complex tasks in complex code ecosystems, never even mention AI, explain that the parser is really generalist-biased, and this article would make perfect sense as an example of exemplary code by a modern c…

Yeah, I think there's a "stone soup" effect going on with AI. It's the same sort of thing you see happening with the customers of psychics. People often have poor awareness of how much they're putting in to a conversation. Or it's a bit like the way Tom Sawyer tricks other kids into painting the fence for him. For me a lot of the magic here is in knowing what questions to ask and when the answers aren't right. If you…

I mostly agree except one critical detail: LLMs are the low code/no code service. You literally tell them what you want and if they’re fine tuned on the problem domain, you’re all set. Microsoft demo’d the office 365 integration and if it works half as well in practice they’ll own the space as much as they have in 1997.

Re: An example of LLM prompting for programming

#180

Post by Nifty3929 »

moyix wrote: post #86
smashface wrote: post #74

Earlier quoted context omitted.

Where do you get that 50% number? Do you mean 50% of all new code in the industry? That seems beyond extremely unlikely.

The number is 40%, and it's 40% of code written by Copilot users . It's also just for Python: > In files where it’s enabled, nearly 40% of code is being written by GitHub Copilot in popular coding languages, like Python—and we expect that to increase. https://github.blog/2022-06-21-github-copilot-is-generally-a...

It's all about the denominator!
Post reply on HN