Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An example of LLM prompting for programming

martinfowler.com

Post reply on HN
Print view View original

211–220 of 297 posts

Re: An example of LLM prompting for programming

#211

Post by peterashford »

ryanjshaw wrote: post #199
afro88 wrote: post #178

The article shows everything that works for this approach. But it's a bit disingenuous. At the end: > Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan. No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of…

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way…

Ooh! That's a really good point - ChatGPT is effectively rubber-ducky as a service =)

Re: An example of LLM prompting for programming

#212

Post by peterashford »

rootusrootus wrote: post #155
gdubs wrote: post #154

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwago…

I agree it's a good idea to take a moderate approach. The hype that LLMs are going to replace SWEs is clearly just that, hype, if you've done any real work trying to get GPT4 to give you the code you want. But it's also clearly a very useful tool. I think it'll absolutely destroy Stack Overflow.

Of course, there's the issue that a lot of the info for useful LLMs probably comes from places like Stack Overflow

Re: An example of LLM prompting for programming

#213

Post by blackbear_ »

yoyohello13 wrote: post #114

I feel like from an information theory perspective there is a lower bound on how little we can write to get a sufficiently specific spec for the AI to generate correct code. This example seems like almost as much work as just writing the code myself. I think English is just too fuzzy, maybe eventually we will get a language tailored to AI that will put more specific limits on the meanings of works. But then how is it…

> a lower bound on how little we can write to get a sufficiently specific spec for the AI to generate correct code.

Interesting though. I believe a lower bound for the number of bits must be at least the log (in base 2) of the probability of such code to appear "in the wild", and larger if the training set is biased and/or the model not fully trained

Re: An example of LLM prompting for programming

#214

Post by LawTalkingGuy »

lcnPylGDnU4H9OF wrote: post #177
svachalek wrote: post #164

Earlier quoted context omitted.

You simply need the system to train itself on its own interactions, like how search engines improve results by counting clicks.

I'm not wondering about how the system will determine what's most helpful but instead determining what's even "correct". A model will learn what's "correct" from Stack Overflow by finding accepted or highly-voted answers but when it can't find such content anymore (in this case because Stack Overflow is hypothetically gone) then what would even exist to generate these discussions to be used as training data? Github,…

Those topics that AI replaces the forums for won't need discussion. People won't be confused about that thing because the coding AI knows the details of it. Soon that'll be most syntax questions, soon simple to mid-level algorithms, etc.

People will move on to higher-level questions.

Re: An example of LLM prompting for programming

#216

Post by hinkley »

What I would really love is if we had a broader linting tool built on this sort of tech that could go the other way.

So often we are halfway through refactoring the code from a bad pattern that has a proven track record of issues, to one that at least prevents the worst ravages of the old one. There are never any guarantees that you will get everyone on board for this. Someone will defect, and they will keep copying and pasting the old pattern and if they code faster than you then you never get to the end.

Give me a way to mark a bunch of code as 'the old way' and hook that information into autocomplete or even just a linter that runs at code review time.

Re: An example of LLM prompting for programming

#217

Post by gumballindie »

drowsspa wrote: post #173
rootusrootus wrote: post #155

Earlier quoted context omitted.

I agree it's a good idea to take a moderate approach. The hype that LLMs are going to replace SWEs is clearly just that, hype, if you've done any real work trying to get GPT4 to give you the code you want. But it's also clearly a very useful tool. I think it'll absolutely destroy Stack Overflow.

Yet, the whole movement of getting blue collar workers to code seems to have lost its steam.

Probably because “graduating” bootcamps doesnt make one a swe and people figured out it’s a scam?

Re: An example of LLM prompting for programming

#218

Post by Fuzzwah »

peterashford wrote: post #211
ryanjshaw wrote: post #199

Earlier quoted context omitted.

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way…

Ooh! That's a really good point - ChatGPT is effectively rubber-ducky as a service =)

This is exactly how I've been explaining LLM tech to my "non-geek" friends and family. I start by explaining rubber ducking, and how I now use chatgpt as a more advanced version of the process.

Re: An example of LLM prompting for programming

#219

Post by hn_throwaway_99 »

ryanjshaw wrote: post #199
afro88 wrote: post #178

The article shows everything that works for this approach. But it's a bit disingenuous. At the end: > Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan. No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of…

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way…

Hmm, I also use ChatGPT as an anti-procrastination tool and task manager, and it's never made a mistake with keeping track of my task list (except that when it sums the estimated times of subgroups of tasks, sometimes those sums are wrong).

Note that it outputs my updated task list every time I add or remove a task (I only asked it to do that one time), so even if old messages go outside of the context window, it's not a big deal because the full updated state of the list is output basically every other message.

Re: An example of LLM prompting for programming

#220

Post by pcthrowaway »

joshribakoff wrote: post #207
pcthrowaway wrote: post #204

I've tried using ChatGPT for writing Vitest tests, and it can't do it, full stop. If you look at the end, it parroted out some tests for jest . True, the APIs are mostly compatible and you can probably change that to Vitest with a couple of lines of code changed, but for more advanced tests, that won't necessarily work. Really disappointed to see this so highly upvoted, when it's pure garbage

That library doesn’t even appear to have a stable release yet, and was at v0.0.x as of a year or so ago… you also may be using chatGPT 3.5 which may predate this library. As a dev with 15 years of experience I haven’t even switched over from jest (but plan to)… all this to say, maybe we can give the bot some slack here. It should be possible to include vitest docs and examples in your prompts to teach it in context,…

Sure, I realize it's unsuccessful at using vitest because it's (relatively) new.

I'm just saying, this was a really telling example of how to use it for prompting.

A very large chunk of the tools I use in Javascript-land are "too new" for ChatGPT to work with properly.

Giving context unfortunately doesn't really work as ChatGPT usually prioritizes what it's absorbed through the corpus over anything you tell it.

To be clear, it does fine with new information if the things you ask it for don't match token sequences it's already been trained on, so if you give it a fictional library and ask it to perform some task with it, that doesn't seem too much like the things it might do with another library that accomplishes a similar thing with a similar API, it will actually use the custom code more successfully.

But for Vitest, it can't accept enough of the docs you might provide for it to be useful to you (though admittedly, sometimes it will show how to do something with jest that at least makes finding the right thing in vitest easier).

By the way, if you are planning to switch over in the future, the path for doing that is seemingly well documented by vitest and seems to be pretty straightforward as well, though I haven't meaningfully used Jest for comparison

edit: to be clear, I'm very impressed with ChatGPT's capabilities, and I think there are good examples of prompting where it does meaningful work in tandem with the human driver exercising their own judgment.

This was an example of a person asking it for things while not pointing out its limitations, which downplays the extent to which one needs to exercise one's judgment when using it. If they failed to point out the things ChatGPT got wrong which I know about, why would I trust that the other things I don't know it got wrong are accurate.

Post reply on HN