Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An example of LLM prompting for programming

martinfowler.com

Post reply on HN
Print view View original

91–100 of 297 posts

Re: An example of LLM prompting for programming

#91

Post by ldhough »

themodelplumber wrote: post #72
Veedrac wrote: post #64

Earlier quoted context omitted.

‘Artists' jobs are safe because AI is bad at hands.’

Artists' jobs are safe in part because they can also use AI, and most already use relevant ecosystems that now incorporate AI. Consumers who can operate AI for clip art purposes are simply still part of the same non-artist-paying demographic they always were. Same with code

Sadly I don't think this is true for art:

https://restofworld.org/2023/ai-image-china-video-game-layof...

I really hope it doesn't end up being the same with code :|

Re: An example of LLM prompting for programming

#92

Post by sanderjd »

twelve40 wrote: post #17
themodelplumber wrote: post #8

If somebody thinks an LLM is coming for everybody's coding job, I'd say this article is a great counterpoint just for existing. You could tell someone from decades ago that we now use a very high level language for complex tasks in complex code ecosystems, never even mention AI, explain that the parser is really generalist-biased, and this article would make perfect sense as an example of exemplary code by a modern c…

Exactly, I actually liked the systematic approach in the article, but it seemed pretty labor-intensive and ... not that much different from other types of programming

To me, that's the whole point of this. I think it is directly analogous to the jump between assembly and higher level compiled languages. You could have said about that, "it still seems pretty labor intensive and not that much different than writing assembly", and that's true, but it was still a big improvement. Similarly, AI-assisted tools haven't solved the "creating software requires work" problem. But I think they're in the process of further shifting the cost curve, making more software possible to make.

Re: An example of LLM prompting for programming

#93

Post by moffkalast »

mjr00 wrote: post #49
cwp wrote: post #45

To me, this is a great illustration of why chat is a terrible interface for a coding tool. I've gone down this path as well, learning that you need to have a detailed prompt that establishes a lot of context, and iteratively improve it to generate better code. And yup, generating a task list and working from that is definitely a key strategy for getting GPT to do anything bigger than a few paragraphs. But compare tha…

Absolutely. People will quickly realize that for coding, the natural language part of LLMs is a distraction. Copilot is much better for someone actually writing code, but unfortunately doesn't get as emphasized due to the narrative surrounding LLMs right now.

Has the Copilot backend been updated to use anything more advanced yet? I tried it out when it was new and free for a while and it really struggled with anything that wasn't incredibly common. GPT 4 in its chat form works a whole lot better for niche stuff than that one did.

Re: An example of LLM prompting for programming

#94

Post by paphillips »

One initial reaction to the prompting style is how similar it is to a human-to-human interaction. For example, a team lead communicating requirements to a wider team composed of less experienced engineers may also follow this type of iterative exchange, continuing until he or she is satisfied that the team understands the work to be done and has the guide rails to be successful.

I recently heard a description about the way this technology will change technical work that resonated: we will become more like the movie director, and less like the actors.

Re: An example of LLM prompting for programming

#95

Post by dpkirchner »

themodelplumber wrote: post #42
mk89 wrote: post #23

For me chatGPT or phind (which is based on chatGPT4, if I understood right) are great documentation tools and also general productivity tools, nothing to say about it. The main issue is that sometimes they really f** it up bad, they make you rethink your knowledge quite deeply (do I remember wrong? did I maybe understand this wrong? is chatGPT wrong?) and this is for me something that can be worse than having to do i…

One thing that really bothers me is that I want it to use best practices and it doesn't really know which ones I'm talking about, and then I realize they are _my_ set of best practices, made from others' nameless best practices. So I have to decide if it's just a matter of manually converting the 5-10 little things like using `env bash` in the header, etc. Or do I ask it to remember that and proceed to the next layer…

> Or do I ask it to remember that and proceed to the next layer of the project

I think this could be solved with a good browser extension. Something that provides an easy to access (e.g., keyboard-only) way to paste customized prompt preludes that enforce your style (or styles if, say, you're using multiple languages).

It looks like Maccy could do the job, albeit not as an extension. I haven't tried it yet.

Re: An example of LLM prompting for programming

#96

Post by agentultra »

nextworddev wrote: post #18
themodelplumber wrote: post #8

If somebody thinks an LLM is coming for everybody's coding job, I'd say this article is a great counterpoint just for existing. You could tell someone from decades ago that we now use a very high level language for complex tasks in complex code ecosystems, never even mention AI, explain that the parser is really generalist-biased, and this article would make perfect sense as an example of exemplary code by a modern c…

The opposite might be true, and here’s why - 1) by using English as spec, the barrier of entry has gone lower, 2) LLMs can also write prompts and self introspect to debug.

But you can't determine if a statement is true by simply reading more words.

It's also not efficient for doing higher level work. There was a time before we had algebra where people were still expressing the same ideas but the notation wasn't there. Mathematics was expressed in "plain language." It's extremely difficult to read for us. For mathematician's of the time there was no other way to explain algorithms or expressions.

For simple programs I have no doubt that these tools enable more people to generate code.

However it's not going to be helpful for people working on hypervisors, networking stacks, operating systems, distributed databases, cryptography, and the like yet. For that you need a more precise language and an LLM that can reason about semantics and generate understandable proofs: not boilerplate proofs either -- they have to be elegant so that a human reading them can understand the problem as well. We're still a ways from being able to do that.

Re: An example of LLM prompting for programming

#97

Post by mov_eax_ecx »

moonchrome wrote: post #57
mov_eax_ecx wrote: post #34

How to overengineer with an LLM, don't state clearly the requirements, shove your pet patterns first, it is more important to follow the slice redux awareness hook than to have working solution, never trust your developers to make decisions, worry more how it is built than building a solution. My way to work with an LLM is to have a good, clear requirement and make the LLM write a possible file organization and query…

How often do you get working code that way ? Unless it's something trivial that fits in it's scope I'd say that's going to produce garbage. I've seen it steer into garbage on longer prompt chains about a single class (of medium complexity) - I doubt it would work project level. Mind sharing the projects ?

I work only with closed source codebases and this approach for prototypes, but, using the same example as the blog i prompt: "the current system is an online whiteboard system. Tech stack: react, use some test framework, use konva for the canvas, propose a file organization, print the file layout tree. (without explanations)." The trick is that for every chat the context is the requirement+the filesystem + the specific file, so you don't have the entire codebase in the context, only the current file, also use gpt4, gpt3 is not good enough.

My main point is that the blog post final output is mock test awareness hook redux, where an architect feels good to see his patterns, with my approach you have a prototype online whiteboard system,

Re: An example of LLM prompting for programming

#98

Post by themodelplumber »

SanderNL wrote: post #89
themodelplumber wrote: post #66

Earlier quoted context omitted.

Ni-dominant. It exists nowadays in various post-Jungian models, many of which are really fascinating, having fleshed it out a lot. The opposing function to Ni is Se, which creates a dichotomy of planning/foreseeing vs. doing/performing. The functions oscillate as a kind of duty cycle, so a lot of sages out there have hobbies as musicians, stage magicians, etc. This dichotomy also effectively shuts out detail memory f…

I couldn't quite tell if you found a beautiful way to insult me, but it is fascinating indeed. I am hand wavey and I understand its failure modes quite well, unfortunately. It's cool to talk about it at this level of abstraction.

No insult intended... I don't really know how much it applies in your case, but since you really took on that viewpoint, that's when the personality theory side of me goes, "well if this is a favored viewpoint then there IS this idea about the population that favors this viewpoint" :-) And thoughts about GPT are generally crafted from general personality positions, in the absence of other relevant self-development experience.

I agree, it's cool stuff

Re: An example of LLM prompting for programming

#99

Post by throwaway202303 »

cwp wrote: post #45

To me, this is a great illustration of why chat is a terrible interface for a coding tool. I've gone down this path as well, learning that you need to have a detailed prompt that establishes a lot of context, and iteratively improve it to generate better code. And yup, generating a task list and working from that is definitely a key strategy for getting GPT to do anything bigger than a few paragraphs. But compare tha…

People have different preferences and habits. Having tried both models I much prefer having a conversation in one window and constructing my code from that in another. Although copilot is about to add some interesting features that may win me back.

Re: An example of LLM prompting for programming

#100

Post by romland »

celeritascelery wrote: post #65
romland wrote: post #9

I started a bit of an exploration around prompts and code a week or three back. I want to figure out the down/up-sides and create tools for myself around it. So, for this project (a game), I decided "for fun" to try to not write any code myself, and avoid narrow prompts that would just feed me single functions for a very specific purpose. The LLM should be responsible for this, not me! It's pretty painful since I sti…

That is really interesting experiment! I have so many questions. - do you feel like this could be a viable work model for real projects? I recognize it will most likely be more effective to balance LLM code with hand written code in the real world. - some of your prompts are really long. Do you feel like the code you get out of the LLM is worth the effort you put in? - given that the code returned is often wrong, do…

I think it could be viable, even right now, with a big caveat, you will want to do some "human" fixes in the code (not just the glue between prompts). The downside of that is you might miss out on parts of the nice natural language story in the commit history. But the upside is you will save a lot of time.

Down the line you will be able to (cheaply) have LLMs know about your entire code-base and at that point, it will definitely become a pretty good option.

On prompt-length, yeah, some of those prompts took a long time to craft. The longer I spend on a prompt, the more variations of the same code I have seen -- I probably get impatient and biased and home in on the exact solution I want to see instead of explaining myself better. When it's gone that far, it's probably not worth it. Very often I should probably also start over on the prompt as it probably can be described differently. That said, if it was in the real world and I was fine with going in and massaging the code fully, quite some time could be saved.

If you don't know how to code, I think it will be very hard. You would at the very least need a lot more patience. But on the flip side, you can ask for explanations of the code that is returned and I must actually say that that is often pretty good -- albeit very verbose in ChatGPT's case. I find it hard to throw a real conclusion out there, but I can say that domain knowledge will always help you. A lot.

I think if you know javascript, you could easily make a game even though you had never ever thought about making a game before. The nice thing about that is that you will probably not do any premature optimization at least :-)

All in all, some prompts was nailed down on first try, the simple particle system was one such example. Some other prompts -- for instance the map-generation with Perlin noise -- might be 50 attempts.

A lot of small decisions are helpful, such as deciding against any external dependencies. It's pretty dodgy to ask for code around some that (e.g. some noise library) that you need to fit into your project. I decided pretty early that there should be no external dependencies at all and all graphics would be procedurally generated. It has helped me as I don't need to understand any libraries I have never used before.

Another note that is related to the above, there are upsides and downsides with high-ish temperature is you get varying results. I think I should probably change my behaviour around that and possibly tweak it depending on how exact I feel my prompt is.

I find myself often wondering where the cap of today's LLM's are, even if we go in the direction of multi-models and have a base which does the reasoning -- and I have to say I keep finding myself getting surprised. I think there is a good possibility that this will be the way some kinds of development will be. But, well, we'd need good local models for that if we work on projects that might be of a sensitive nature.

Related to amount of prompt attempts: I think the game has cost me around $6 in OpenAI fees so far.

One particularly irritating (time consuming) prompt was getting animated legs and feet: https://github.com/romland/llemmings/commit/e9852a353f89c217...

Post reply on HN