Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Prompt-Engineer

github.com

Post reply on HN
Print view View original

71–80 of 166 posts

Re: GPT-Prompt-Engineer

#71

Post by larodi »

fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

it seems to me also, that this is very much some sort of snake oil for the llm era. prompt generation varies from llm-to-llm and I doubt gpt4 can do reasonable evaluation, provided that it does not know at all about other models.

Re: GPT-Prompt-Engineer

#72

Post by jstanley »

donkeyboy wrote: post #67
fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

There is a paper on arxiv saying that GPT4 correlation with human evaluators on a variety of tasks with strongly positive. I am also uncomfortable with it, but using GPT4 as a grader is not as bad as you think.

That's definitely begging the question.

If you're prepared to accept that GPT-4 can answer questions just as well as humans can, why do you even need to do prompt engineering?

Re: GPT-Prompt-Engineer

#73

Post by toxicFork »

fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

Well, you could keep everything else in the project and put yourself or a human as the "does this result feel better than the other one" decision maker

Re: GPT-Prompt-Engineer

#74

Post by larodi »

brunoluiz wrote: post #34

Isn’t engineering an exact science while prompt engineering is completely not? Although, even software engineering being an exact science, it is a funny one: most of us don’t get certified as like, let’s say, mechanical engineers do. Would they say we are engineers? So perhaps the “engineer” term got overloaded in recent years?

you make a valid point, and no - we are not engineers. we are people with printed labels at best, where the label says architect or engineer. but most of these people with these labels don't even have a degree, which is the prerequisite to have this designations. architects also typically need to comply for a local guild.

we, the IT crowd, are long over-due for this formalization of the professions.

Re: GPT-Prompt-Engineer

#75

Post by l5870uoo9y »

Apfel wrote: post #57

Usage query: It looks like this could get expensive quite quickly. The approach is great, but with GPT-4 especially, could be very difficult. Is it worth using with 3.5 as a first pass then switching prompts to GPT4 once you've got the best prompt?

I am not sure this approach is doable since GPT-4 is capable of solving assignments that GPT-3.5 get wrong. Example GPT-3.5 fails to solve the prompt (with the dvdrental sample database schema added [1]):

> find customers who didn't rent a movie in the last 12 months but rented a movie in the 12 months before that

GPT-4 solves this without a problem [2]. Combining logic like (without additional database schema added):

> find all users who lives in Paris using lat/lng and who visited the south of France within the last month

GPT-3.5 can't understand this at all, GPT-4 solves it [3].

[1]: https://www.postgresqltutorial.com/postgresql-getting-starte...

[2]: https://aihelperbot.com/snippets/cljy8km2h0000my0fgq8kut5w

[3]: https://aihelperbot.com/snippets/cljy8q6gz000al70fvfzxt2hh

Re: GPT-Prompt-Engineer

#76

Post by londons_explore »

larodi wrote: post #71
fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

it seems to me also, that this is very much some sort of snake oil for the llm era. prompt generation varies from llm-to-llm and I doubt gpt4 can do reasonable evaluation, provided that it does not know at all about other models.

The various leaderboards show that, in aggregate, LLM's acting as evaluators match other LLM's and humans remarkably closely.

Re: GPT-Prompt-Engineer

#77

Post by LASR »

donkeyboy wrote: post #67
fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

There is a paper on arxiv saying that GPT4 correlation with human evaluators on a variety of tasks with strongly positive. I am also uncomfortable with it, but using GPT4 as a grader is not as bad as you think.

I think the parent poster is saying that it’s grading the prompts and not the output generated from the prompts.

Yeah I agree there. Unless you can check against the output, it’s not really telling much.

Re: GPT-Prompt-Engineer

#78

Post by ris58h »

School-Cotton wrote: post #60
brunoluiz wrote: post #34

Isn’t engineering an exact science while prompt engineering is completely not? Although, even software engineering being an exact science, it is a funny one: most of us don’t get certified as like, let’s say, mechanical engineers do. Would they say we are engineers? So perhaps the “engineer” term got overloaded in recent years?

“Engineer” originally just meant someone who builds engines (in a broad sense of that word). The formal titles requiring certification, etc. are the more recent development.

https://en.wikipedia.org/wiki/Engineer

> The word engineer (Latin ingeniator) is derived from the Latin words ingeniare ("to contrive, devise") and ingenium ("cleverness").

Re: GPT-Prompt-Engineer

#79

Post by hospitalJail »

skwirl wrote: post #59
mellosouls wrote: post #40

Earlier quoted context omitted.

Software engineer has always been a daft, grandiose term that seems aimed at prestige rather than reality in the majority of cases. If coders can call themselves engineers, no reason why anybody else solving puzzles for a living can't.

I’m guessing you aren’t from the US because in the US there is not much prestige in the title of engineer, but it seems to be a title people in Europe get weirdly hung up on. If anything, software engineering positions are more prestigious than most traditional engineering positions here.

I was a real engineer for a decade before switching to programming.

I cannot use the word software engineer, since its nothing like real engineering.

Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and it costs 1M to redo the tool, but it also costs 16 weeks to redo the tool. We'd even offer to pay absurd money, prevent future business, etc... to get the tool done in 8 weeks, but its impossible to get it done faster. Now everything in the company is 8 weeks behind schedule.

Anyway, real engineering was harder, but programming pays soooo much more money. Its a demand thing, not a difficulty thing.

Re: GPT-Prompt-Engineer

#80

Post by hamasho »

Off topic but Jupyter cells on GitHub can't display horizontally long content and it frustrates me a lot. This small piece of code for the browser console helps me see more content, but it only works in large displays.

    $("[data-type='ipynb']").style.width = '100%'
Post reply on HN