Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An example of LLM prompting for programming

martinfowler.com

Post reply on HN
Print view View original

201–210 of 297 posts

Re: An example of LLM prompting for programming

#201

Post by alexashka »

gdubs wrote: post #154

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwago…

You may be underestimating how much meaning people derive from jumping on bandwagons and having a simple to understand group identity.

Your suggestion would make many people unhappy. They can't win the competence game and hence 'really investigating' is a losing proposition for them. What they can do is jump on bandwagons very quickly, hoping to score a first mover advantage.

How much of an advantage would one get from taking a couple of years to really investigate Bitcoin and the algorithms involved, vs buying some as early as possible and telling everyone else how great it is? :)

Re: An example of LLM prompting for programming

#202

Post by simonw »

AzzieElbab wrote: post #194
afro88 wrote: post #192

Earlier quoted context omitted.

Are you using the GPT4 model? There's a very significant improvement between 3.5 (the free one) and 4.

I am supposedly on GPT4 via GPT+. I try using it for boilerplatey things, like terraform, and the results are simply incorrect. It seems more helpful in providing examples, even for some far more complex tech - like rust code.

Does it say GPT-4 at the top of the screen?

Re: An example of LLM prompting for programming

#203

Post by all2 »

mrbonner wrote: post #136

I can’t help but thinking: this is way more work than just code it myself? Anybody has the same thought?

It depends how you use it. I've been using it to skip boilerplate coding and get straight to the meaty bits. It took me a few days to sketch out an application using ChatGPT to handle the boilerplate, including dependency management (python, poetry, etc.).

I've had to handle the specific pieces of implementation myself. Especially unit testing new pieces of code. When asked to generate unit tests, it does ok, but it doesn't get the spirit of the code (my intended purpose) and so I'm left filling in a bunch of blanks.

Re: An example of LLM prompting for programming

#204

Post by pcthrowaway »

I've tried using ChatGPT for writing Vitest tests, and it can't do it, full stop.

If you look at the end, it parroted out some tests for jest. True, the APIs are mostly compatible and you can probably change that to Vitest with a couple of lines of code changed, but for more advanced tests, that won't necessarily work.

Really disappointed to see this so highly upvoted, when it's pure garbage

Re: An example of LLM prompting for programming

#205

Post by DonHopkins »

mpaepper wrote: post #150
1024core wrote: post #104

For some reason, this reminds me of how we used to give instructions to Indian coders in the 90s and early 2000s. You would have to spell out everything. What you got back was nearly there, but some back-and-forth was involved. This brings back some terrible memories.

The big difference is that you get the results immediately and iterations take minutes not days

And no time zone differences!

Re: An example of LLM prompting for programming

#206

Post by scarface74 »

mjr00 wrote: post #200
scarface74 wrote: post #197

Earlier quoted context omitted.

Isn’t that the game changer though that you can use natural language and treat it like the “worlds smartest intern” and I can just give it the list of my requirements? It’s the difference between: “Python script to return all of the roles with a given policy AWS” (answer found on StackOverflow with Google) And with ChatGPT “Write a Python script that returns AWS IAM roles that contain one or more policies specified b…

> “Write a Python script that returns AWS IAM roles that contain one or more policies specified by one or more -p arguments. Use argparse to accept parameters and output the found roles as a comma separated list” Again, this is completely unnecessary. This is like in the old days when technically illiterate people would quite literally Ask Jeeves[0] and search for full questions because they didn't know how to interf…

At first, when I didn’t specify “use argparse” it would use raw argument parsing

It also thought I actually wanted a file called “output.csv” based on your text and gave me an actual argument to specify the output file that I didn’t want.

There is a lot of nuance to my requirements that ChatGPT missed with your keywords.

Sidenote: there is a bug in both versions and also when I did this for real. Most AWS list APIs use pagination. You have to tell it that “this won’t work with more than 50 roles” and it will fix it.

Re: An example of LLM prompting for programming

#207

Post by joshribakoff »

pcthrowaway wrote: post #204

I've tried using ChatGPT for writing Vitest tests, and it can't do it, full stop. If you look at the end, it parroted out some tests for jest . True, the APIs are mostly compatible and you can probably change that to Vitest with a couple of lines of code changed, but for more advanced tests, that won't necessarily work. Really disappointed to see this so highly upvoted, when it's pure garbage

That library doesn’t even appear to have a stable release yet, and was at v0.0.x as of a year or so ago… you also may be using chatGPT 3.5 which may predate this library. As a dev with 15 years of experience I haven’t even switched over from jest (but plan to)… all this to say, maybe we can give the bot some slack here. It should be possible to include vitest docs and examples in your prompts to teach it in context, did you try that?

Re: An example of LLM prompting for programming

#208

Post by MacsHeadroom »

nuancebydefault wrote: post #196
tarruda wrote: post #190

Earlier quoted context omitted.

> The hype that LLMs are going to replace SWEs is clearly just that, hype LLMs cannot replace anyone, but it is clear that engineers which master LLMs usage might multiply their productivity by a lot. The question is: If one LLM assisted engineer can work 10x faster, will companies reduce their engineer staff by 90%?

To answer your question with a question if I may -- when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain.

>when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain.

The average AI company has like 1 employee per $25M valuation. That's around 25x fewer employees than the typical tech company.

Re: An example of LLM prompting for programming

#209

Post by travisjungroth »

ryanjshaw wrote: post #199
afro88 wrote: post #178

The article shows everything that works for this approach. But it's a bit disingenuous. At the end: > Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan. No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of…

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way…

Good rule of thumb with ChatGPT: you can’t exit loops. Once you’ve gone A > B > A, your best move is to start a new chat. Even then it may reproduce and you should do some similar but different task. Remember that it’s a prediction engine, weighing heavily on the existing prompt. So you say B again, or B1 and it’s like, I know what to do! A! Cause last time was A->B so let’s do it again.

In your case this would be “[]Task1”, “Task1 is done”, “[]Task1”, [here is where you start a new chat or fix it yourself if possible].

Re: An example of LLM prompting for programming

#210

Post by joseph_grobbles »

gdubs wrote: post #154

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwago…

[dead]
Post reply on HN