Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Prompt-Engineer

github.com

Post reply on HN
Print view View original

91–100 of 166 posts

Re: GPT-Prompt-Engineer

#91

Post by kossTKR »

muzani wrote: post #15
mangecoeur wrote: post #11

Should we really call it "engineering" if its a case of "try random things until one of them works without really knowing why"?

Looks more like science here. Run a bunch of experiments and see which does better. But literally the first sentence of the readme is "Prompt engineering is kind of like alchemy."

The difference is that in science the dialectic is between you the observer or a group of observers and a "reality" that hopefully(!) remains relatively stable, because the human organism is the way that it is - just as physical reality.

But these models are changing literally every day, so there's no fixed thing to reveal.

So no one will be able to reproduce anything at all.

This makes all this "engineering" pretty ridiculous in my eyes, it's literally for one models bizarre emergent properties.

Re: GPT-Prompt-Engineer

#92

Post by [deleted] »

bravura wrote: post #64
quickthrower2 wrote: post #30

Earlier quoted context omitted.

I think it is a freaking miracle it even works. I understand how it works for say 100 parameter linear regression, but that it would work for billions of parameters (billion dimensional space) by nudging the parameters each a little (based on purely it's impact on the loss, assuming everything stays the same), is not obvious to me. It is a kind of magic. Regarding randomness, the initialization of the weights is rand…

To be honest, there’s always been a tension in machine learning between the: “we won’t do it unless the theory is complete and sound” versus the “we don’t understand this but it works much better consistently so we do it” crowd. In the 90s and even 00s, the theory first crowd was mainstream and the empirical first crowd was considered fringe. Very fringe. Personally I appreciated it when LeCun was like: “you can’t fi…

[deleted]

Re: GPT-Prompt-Engineer

#93

Post by SirMaster »

hospitalJail wrote: post #79
skwirl wrote: post #59

Earlier quoted context omitted.

I’m guessing you aren’t from the US because in the US there is not much prestige in the title of engineer, but it seems to be a title people in Europe get weirdly hung up on. If anything, software engineering positions are more prestigious than most traditional engineering positions here.

I was a real engineer for a decade before switching to programming. I cannot use the word software engineer, since its nothing like real engineering. Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and…

So if you have your PE license and are writing software that controls a commercial airplane system or a spacex rocket, you are not an engineer doing software engineering?

Re: GPT-Prompt-Engineer

#94

Post by aiisjustanif »

larodi wrote: post #74
brunoluiz wrote: post #34

Isn’t engineering an exact science while prompt engineering is completely not? Although, even software engineering being an exact science, it is a funny one: most of us don’t get certified as like, let’s say, mechanical engineers do. Would they say we are engineers? So perhaps the “engineer” term got overloaded in recent years?

you make a valid point, and no - we are not engineers. we are people with printed labels at best, where the label says architect or engineer. but most of these people with these labels don't even have a degree, which is the prerequisite to have this designations. architects also typically need to comply for a local guild. we, the IT crowd, are long over-due for this formalization of the professions.

I feel like many places have tried this and didn’t like it or molded it into a Frankenstein.

The idea of an engineer that researches, designs, tests, and measures and a programmer that implements seemed to cost too much (not just monetarily) for the industry that employees them and there isn’t the sufficient need to regulate all sun-groups of the industries that employ software programmers / engineers.

Re: GPT-Prompt-Engineer

#95

Post by hospitalJail »

SirMaster wrote: post #93
hospitalJail wrote: post #79

Earlier quoted context omitted.

I was a real engineer for a decade before switching to programming. I cannot use the word software engineer, since its nothing like real engineering. Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and…

So if you have your PE license and are writing software that controls a commercial airplane system or a spacex rocket, you are not an engineer doing software engineering?

Ahh I forgot to put my usual caveat.

Assembly and safety critical C can be considered software engineering.

Anything with abstraction, no.

Re: GPT-Prompt-Engineer

#96

Post by jes »

tough wrote: post #84
xmcqdpt2 wrote: post #43

Earlier quoted context omitted.

In my opinion, coding is a craft. As software has only existed for like 70 years, we are more like the guilds building cathedrals in the middle ages than like modern civil engineering. One day I think there will be true software engineering. When that happens you won't be able to start software projects without certifications, and most people (or programs!) who actually do the coding will be following careful plans a…

> One day I think there will be true software engineering. When that happens you won't be able to start software projects without certifications, and most people (or programs!) who actually do the coding will be following careful plans and instructions from the engineers who designed the project. Sounds really bad. Software can't fall on your head and kill you, not all of it at least. Different software should requir…

You might enjoy this article on the Therac-25 [1]. It's kind of the standard example of how errors in software can wind up harming people. I have written medical device software for about 30 years. In my experience, delivering high quality software for Class B and Class C devices is both challenging and expensive.

https://en.wikipedia.org/wiki/Therac-25

Re: GPT-Prompt-Engineer

#97

Post by judge2020 »

namaria wrote: post #33
muzani wrote: post #15

Earlier quoted context omitted.

Looks more like science here. Run a bunch of experiments and see which does better. But literally the first sentence of the readme is "Prompt engineering is kind of like alchemy."

> Run a bunch of experiments and see which does better. That's empiricism. Scientific method implies formulating falsifiable theoretical claims, producing meaningful analysis of experimental conditions, publishing peer-reviewed and reproducible results.

That's if you're trying to become a scientist. You can do science even if your experiment doesn't contain any of those elements besides trying multiple things and observing the outcomes.

Re: GPT-Prompt-Engineer

#98

Post by 9659 »

SirMaster wrote: post #93
hospitalJail wrote: post #79

Earlier quoted context omitted.

I was a real engineer for a decade before switching to programming. I cannot use the word software engineer, since its nothing like real engineering. Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and…

So if you have your PE license and are writing software that controls a commercial airplane system or a spacex rocket, you are not an engineer doing software engineering?

From Wikipedia: "A software engineer is a person who applies the engineering design process to design, develop, maintain, test, and evaluate computer software."

"The engineering design process, also known as the engineering method, is a common series of steps that engineers use in creating functional products and processes. The process is highly iterative - parts of the process often need to be repeated many times before another can be entered - though the part(s) that get iterated and the number of such cycles in any given project may vary.

It is a decision making process (often iterative) in which the basic sciences, mathematics, and engineering sciences are applied to convert resources optimally to meet a stated objective. Among the fundamental elements of the design process are the establishment of objectives and criteria, synthesis, analysis, construction, testing and evaluation.[1]"

It is not dependent on the problem domain, rather on how the work is performed.

Re: GPT-Prompt-Engineer

#99

Post by philovivero »

hospitalJail wrote: post #79
skwirl wrote: post #59

Earlier quoted context omitted.

I’m guessing you aren’t from the US because in the US there is not much prestige in the title of engineer, but it seems to be a title people in Europe get weirdly hung up on. If anything, software engineering positions are more prestigious than most traditional engineering positions here.

I was a real engineer for a decade before switching to programming. I cannot use the word software engineer, since its nothing like real engineering. Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and…

> While many software problems can cause you to lose money, engineering problems can cause you to lose time.

I'm guessing you were trying to say something else here. I literally cannot think of a single software engineering problem I've ever encountered that didn't cost time. By your definition, then, software engineering is engineering. Your claim and your definitions are at odds with one-another.

Also, you don't directly claim it, but you seem to imply, that software engineering can't have real-world consequences... or something? As another reply points out, sometimes software is in the critical path for things like rockets and airplanes, where mistakes cost lives.

And some people making software for less life-altering systems take their craft just as seriously. Some people think that losing $10M every single second while their software is failing is a big deal.

Are you claiming people who write HFT code, ad arbitrage code, code that powers the front page of Apple, Amazon, Microsoft, and Google are just cowboying it through the day, doing nothing special?

Overall I just find this comment very confused. Maybe you could put some thought into what you're trying to say, and say it better?

Re: GPT-Prompt-Engineer

#100

Post by otikik »

donkeyboy wrote: post #67
fatso784 wrote: post #65

This tool doesn’t benchmark based on how a model actually responds to the generated prompts. Instead, it trusts GPT4 to rank prompts simply in terms of how well it imagines they will perform head-to-head. Thus, there’s no way to tell if the chosen ‘best prompt’ actually is the best, because there’s no ground truth against actual responses. Why is this so popular, then (more popular than promptfoo, which I think is a…

There is a paper on arxiv saying that GPT4 correlation with human evaluators on a variety of tasks with strongly positive. I am also uncomfortable with it, but using GPT4 as a grader is not as bad as you think.

Of course there is strong correlation. That is literally what it was designed to do.

The problem is that it will simultaneously say that "cow eggs are bigger than chicken eggs", with the same confidence (and in a way that correlates well with human evaluators).

https://www.reddit.com/r/Funnymemes/comments/10ohd2n/chatgpt...

So when you get an evaluation you are playing the russian roulette - you may get a decent result, or you may get cow eggs.

Post reply on HN