An example of LLM prompting for programming • Hacker News Board

Re: An example of LLM prompting for programming

#51

Post by romland » Tue, Apr 18, 2023, 1:35 PM UTC

sk0g wrote: post #27

romland wrote: post #9

I started a bit of an exploration around prompts and code a week or three back. I want to figure out the down/up-sides and create tools for myself around it. So, for this project (a game), I decided "for fun" to try to not write any code myself, and avoid narrow prompts that would just feed me single functions for a very specific purpose. The LLM should be responsible for this, not me! It's pretty painful since I sti…

That's a beautiful readme, starred! Out of curiosity, right now would you say you have saved time by (almost) exclusively prompting instead of typing the code up yourself? Do you see that trending in another direction as the project progresses?

It was far easier to get a big chunks of work done in the beginning, but that is pretty much how it works for a human too (at least for me). The thing that limit you is the context-length limit of the LLM, so you have to be rather picky on what existing code you feed back in. With this then comes the issue with all the glue between the prompts, so I can see that the more polished things will need to become, the more human intervention -- this is a trend I already very much see.

If there is time saved, it is mostly because I don't fear some upcoming grunt work. Say, for instance, creating the "Builder" lemming. You know pretty much exactly how to do it but you know there will be a lot of one-off errors and subtle issues. It's easier to go at it by throwing together some prompt a bit half-heartedly and see where it goes.

On some prompts, several hours were spent, mostly reading and debugging outputs from the LLM. This is where it eventually gets a bit dubious -- I now know pretty much exactly how I want the code to look since I have seen so many variants. I might find myself massaging the prompt to narrow in on my exact solution instead of making the LLM "understand the problem".

Much of this is due to the contrived situation (human should write little code) -- in the real world you would just fix the code instead of the prompt and save a lot of time.

Thank you, by the way! I always find it scary to share links to projects! :-)

Re: An example of LLM prompting for programming

#52

Post by romland » Tue, Apr 18, 2023, 1:36 PM UTC

ChatGTP wrote: post #26

romland wrote: post #9

I started a bit of an exploration around prompts and code a week or three back. I want to figure out the down/up-sides and create tools for myself around it. So, for this project (a game), I decided "for fun" to try to not write any code myself, and avoid narrow prompts that would just feed me single functions for a very specific purpose. The LLM should be responsible for this, not me! It's pretty painful since I sti…

Just curious, you’re using which version?

I have experimented quite a bit with various flavours of LLaMa, but have had little success in actually getting not-narrow outputs out of them.

Most of the code in there now is generated by gpt-3.5-turbo. Some commits are by GPT-4, and that is mostly due to context length limitations. I have tried to put which LLM was used in every non-human commit, but I might have missed it in some.

Re: An example of LLM prompting for programming

#53

Post by maroonblazer » Tue, Apr 18, 2023, 1:37 PM UTC

cwp wrote: post #45

To me, this is a great illustration of why chat is a terrible interface for a coding tool. I've gone down this path as well, learning that you need to have a detailed prompt that establishes a lot of context, and iteratively improve it to generate better code. And yup, generating a task list and working from that is definitely a key strategy for getting GPT to do anything bigger than a few paragraphs. But compare tha…

As a hobbyist developer with no formal training, I wish Copilot had a 'teaching' or "Senior Dev" mode, where I can play the role of the Junior Dev. I'd like it to pick up on what I'm trying to write, and then prompt me with questions or hints, but not straight up give me the code.

Or, if that's too Clippy-like annoying, let me prompt it when I'm stuck, and only then suggest hints or ask suggestive questions that guide me to a solution.

I agree, very exciting to see where all this goes.

Re: An example of LLM prompting for programming

#54

Post by moonchrome » Tue, Apr 18, 2023, 1:37 PM UTC

mk89 wrote: post #23

For me chatGPT or phind (which is based on chatGPT4, if I understood right) are great documentation tools and also general productivity tools, nothing to say about it. The main issue is that sometimes they really f** it up bad, they make you rethink your knowledge quite deeply (do I remember wrong? did I maybe understand this wrong? is chatGPT wrong?) and this is for me something that can be worse than having to do i…

Honestly to me it happens more than it doesn't - but maybe that's because I've tried it in cases where I've already used traditional approaches to come up with the answer and going to GPT and phind to benchmark their viability.

I've mentioned it on other thread, but phind's "google-fu" is weak, it does a shallow pass and bing index (I'm assuming) is worse than google. It's also slow as hell with GPT4 which makes digging deeper slower than just manually going in.

Re: An example of LLM prompting for programming

#55

Post by SamPatt » Tue, Apr 18, 2023, 1:37 PM UTC

cwp wrote: post #45

To me, this is a great illustration of why chat is a terrible interface for a coding tool. I've gone down this path as well, learning that you need to have a detailed prompt that establishes a lot of context, and iteratively improve it to generate better code. And yup, generating a task list and working from that is definitely a key strategy for getting GPT to do anything bigger than a few paragraphs. But compare tha…

I've noticed that after using copilot on a code base for a while, you can effectively prompt the AI just by creating a descriptive comment.

// This function ends the call by sending a disconnection message to all connected peers

Bam, copilot will recommend at least the first line, with subsequent lines usually being pretty good, and more and more frequently, it will recommend the whole function.

I still use GPT-4 a lot, especially for troubleshooting errors, but I'm always pleasantly surprised at how good copilot can be.

Re: An example of LLM prompting for programming

#56

Post by mjr00 » Tue, Apr 18, 2023, 1:37 PM UTC

yodsanklai wrote: post #50

mjr00 wrote: post #49

Earlier quoted context omitted.

Absolutely. People will quickly realize that for coding, the natural language part of LLMs is a distraction. Copilot is much better for someone actually writing code, but unfortunately doesn't get as emphasized due to the narrative surrounding LLMs right now.

> Copilot is much better for someone actually writing code I haven't used copilot yet, but I'm using occasionally chatgpt with prompts such as "write a bash/python script take takes these parameters and perform this tasks". Then I iterate if needed, and usually, i can get what i want faster than without using chatgpt. It's not a game changer, but it's a performance boost. How natural language is a distraction here? a…

Try not using natural language and just type what you'd type into Google. You'll get the same results and realize that all of the natural language fluff is totally unnecessary. I just typed in "bash script recursive chmod 777 all files" (as a dumb toy example) and got a resulting script back. It was surrounded by two natural language GPT comments:

> It's generally not recommended to give all files and directories the 777 permission as it can pose a security risk. However, if you still want to proceed with this, here's a bash script that recursively changes the permission of all files and directories to 777: [...] Make sure to replace "/path/to/target/directory" with the path of the directory you want to modify. To run the script, save it as a file (e.g., "chmod_all.sh"), make it executable with the command "chmod +x chmod_all.sh", and then run it with "./chmod_all.sh".

It's up to the reader to decide if those are necessary, but I'd lean towards no.

Re: An example of LLM prompting for programming

#57

Post by moonchrome » Tue, Apr 18, 2023, 1:39 PM UTC

mov_eax_ecx wrote: post #34

How to overengineer with an LLM, don't state clearly the requirements, shove your pet patterns first, it is more important to follow the slice redux awareness hook than to have working solution, never trust your developers to make decisions, worry more how it is built than building a solution. My way to work with an LLM is to have a good, clear requirement and make the LLM write a possible file organization and query…

How often do you get working code that way ? Unless it's something trivial that fits in it's scope I'd say that's going to produce garbage. I've seen it steer into garbage on longer prompt chains about a single class (of medium complexity) - I doubt it would work project level. Mind sharing the projects ?

Re: An example of LLM prompting for programming

#58

Post by mason55 » Tue, Apr 18, 2023, 1:39 PM UTC

themodelplumber wrote: post #42

mk89 wrote: post #23

For me chatGPT or phind (which is based on chatGPT4, if I understood right) are great documentation tools and also general productivity tools, nothing to say about it. The main issue is that sometimes they really f** it up bad, they make you rethink your knowledge quite deeply (do I remember wrong? did I maybe understand this wrong? is chatGPT wrong?) and this is for me something that can be worse than having to do i…

One thing that really bothers me is that I want it to use best practices and it doesn't really know which ones I'm talking about, and then I realize they are _my_ set of best practices, made from others' nameless best practices. So I have to decide if it's just a matter of manually converting the 5-10 little things like using `env bash` in the header, etc. Or do I ask it to remember that and proceed to the next layer…

I think that Copilot is much better/more promising for this kind of thing because it's looking at the code you've already written without you having to constantly prompt it.

I had a lot of the same hangups as you when I had played around with ChatGPT. How do I get it to handle the monotonous stuff without me having to spend all my time teaching it?

I finally tried Copilot the other day and it was stunning. I had a half-written golang client that was a wrapper around an undocumented and poorly structured API for a tool we use. I had written the get and create methods. Then I added a comment with an example URL for delete and Copilot auto-completed the entire method in the same style as the two methods I had already written. In some cases, like formatting & error handling, it was exactly the same as what I'd written, but other cases, like variable naming, string templating, etc., it replicated the spirit of my style but adapted for this new "delete" method.

I think ChatGPT is just the wrong interface for this kind of thing (at least right now).

Re: An example of LLM prompting for programming

#59

Post by SanderNL » Tue, Apr 18, 2023, 1:42 PM UTC

themodelplumber wrote: post #48

SanderNL wrote: post #32

Earlier quoted context omitted.

Fair enough, but don’t we abstract “upwards” all the time? Assembly won’t go away, but do you deal with it?

For one, assembly ceases to be a relevant detail and is replaced by other relevant details. So, I can't code fast games in a 1984 workplace, currently, being too out of touch with assembly on a given chipset. But I also can't wave my hands at an LLM and expect a modern, fast game of the desired quality to code itself. (Even though a clip art-style result is possible, the requirements are always going to be special de…

Now you got me on the edge of my seat. What is this personality type?

Re: An example of LLM prompting for programming

#60

Post by moonchrome » Tue, Apr 18, 2023, 1:42 PM UTC

I feel like this is a bunch of ceremony and back and forth, and also considering GPT-4 speed - I feel like I would fly past this approach just using copilot and coding.

I look forward to offloading these kinds of tasks to LLMs but I'm not seeing the value right now. Using them feels slow and unsatisfying, need to triple check everything, specify everything relevant for context.

Also maybe it's just me but verbalizing requirements unambiguously can often be harder than writing code for it. And it's not fun. If GPT4 was GPT3.5 fast it would probably be a completely different story.