Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sorry, but a new prompt for GPT-4 is not a paper

twitter.com

Post reply on HN
Print view View original

51–60 of 193 posts

Re: Sorry, but a new prompt for GPT-4 is not a paper

#51

Post by carbocation »

The art and science of building these models is not disputed, but I think that the scientific value of prompts is tightly linked to reproducibility.

If you’ve developed a new prompt for a model whose weights you can directly access, then this prompt could have scientific value because its utility will not diminish over time or be erased with a new model update. I’m even generally of the view that a closed API endpoint whose expiration date is years into the future could have some value (but much less so). But simply finding a prompt for something like ChatGPT is not useful for science because we don’t even have certainty about which model it’s executing against.

Note that some of the best uses of these models and prompting have nothing to do with academics; this is a comment focused on the idea about writing academic papers about prompts.

Re: Sorry, but a new prompt for GPT-4 is not a paper

#52

Post by azinman2 »

thesz wrote: post #45
selfhoster11 wrote: post #22

Excuse me? Step by step wasn't paper-worthy? Hard disagree. LLM research is currently in its infancy, because they are no older than a few years old. And a research field in its infancy is bound to have a few noteworthy "no sh*t, Sherlock" papers that would be obvious from hindsight. The fact is, LLMs are a higher-order construct in machine learning, much like a fish is higher-order than a simple cellular colony. Low…

> Lower-order ML constructs do not demonstrate emergent capabilities like step by step, stream of consciousness thinking, and so on. As a matter of fact, I did a project on the normalization of the text, e.g., translate "crossing of 6 a. and 12 s." into "crossing of sixth avenue and 12-th street" with a simple LM (order 3) and beam search on the lattice paths, lattice formed with hypotheses' variants. I got two fold…

Is there something you can link to? I’d like to learn more.

Re: Sorry, but a new prompt for GPT-4 is not a paper

#54

Post by eimrine »

dartos wrote: post #46
phkahler wrote: post #38

Earlier quoted context omitted.

>> LLM research is currently in its infancy Everything that went in to creating GPT4 is AI/science or whatever. Probing GPT4 and trying to understand and characterize it is also a very worthy thing to do - else how can it be improved upon? But if making GPT is science, I'd say this stuff is more akin to psychology ;-)

What is psychology, but applied biology? https://xkcd.com/435/

Psychology is a religious-like pseudoscience, they can not even define what "psy" is without using some conceptions from religions such as a soul.

upd these statements from me are so controversial, the number of "points" just dances lambada. The psy* areas are clearly polarized: some guys upvote all my messages in this topic and some other ones downvote all my messages in this topic. This is a sign of something interesting but I am not ready to elaborate on this statement in this comment which is going to become [flagged] eventually.

Re: Sorry, but a new prompt for GPT-4 is not a paper

#55

Post by skilled »

I can maybe understand the frustration from a “scientific” perspective, but for a lot of these “one prompt papers” - you still need someone to sit down and do the analysis and comparisons. Very few papers focus only on GPT/ChatGPT.

Additionally, it gives people other ideas to try for themselves. And some of this stuff might be useful to someone in a specific scenario.

It’s not glamorous research or even future-proof seeing as how certain prompts can be surgically removed or blocked by the owner of the model, but I don’t think it warrants telling people not to do it.

Re: Sorry, but a new prompt for GPT-4 is not a paper

#56

Post by Woshiwuja »

eimrine wrote: post #54
dartos wrote: post #46

Earlier quoted context omitted.

What is psychology, but applied biology? https://xkcd.com/435/

Psychology is a religious-like pseudoscience, they can not even define what "psy" is without using some conceptions from religions such as a soul. upd these statements from me are so controversial, the number of "points" just dances lambada. The psy* areas are clearly polarized: some guys upvote all my messages in this topic and some other ones downvote all my messages in this topic. This is a sign of something inter…

What? Are you literally referring to ancient greek psy? jesus...

Re: Sorry, but a new prompt for GPT-4 is not a paper

#58

Post by jhbadger »

ilc wrote: post #14
3cats-in-a-coat wrote: post #5

Attacking the participants in a systemic shift is 100% useless as it doesn't target the culprit. In programming we have a similar phenomenon, that StackOverflow-driven (and I guess now GPT-driven) juniors have overtaken the industry and displaced serious talent. Because sufficient amounts of quantity always beats quality, even if the end result is inferior, this is caused by market dynamics, which operate on much cru…

This whole thing reminds me a bit of playing Fallout 4. There will be good data, the pre-AI enshitification data. The stuff from before the war. And then... the data after. Tainted by the entropy, and lack of utility of AI. Alas, this means in some senses, human progress will slow and stop in the tech field if we aren't careful and preserve ways to create pre-AI data. But the cost of it is so high in comparison to po…

This paints a rosy picture of human-generated data now. It's not as if most human data is reliable. Even among peer reviewed scientific literature, most of it is crap and it takes effort to find the good stuff. Also, your analogy kind of misses the point of the Fallout games. The pre-war world was awful, filled with evil corporations like Vault-tech and Nuka Cola that murdered and poisoned their customers, and the point of the games is that people need to move on and not idealize the past.

Re: Sorry, but a new prompt for GPT-4 is not a paper

#59

Post by jdefr89 »

elif wrote: post #42

Nah, this is just an early example of many "this is too easy it doesn't count" defensive human arguments against AI. Parallel to the "you use copilot so your code quality is terrible and you don't really even understand it so it's not maintainable" human coping we are familiar with. If there is any shred of truth to these defenses, it is temporary and will be shown false by future, more powerful AI models. Consider t…

Id much rather see a PoC…

Re: Sorry, but a new prompt for GPT-4 is not a paper

#60

Post by idiliv »

Der_Einzige wrote: post #26
selfhoster11 wrote: post #22

Excuse me? Step by step wasn't paper-worthy? Hard disagree. LLM research is currently in its infancy, because they are no older than a few years old. And a research field in its infancy is bound to have a few noteworthy "no sh*t, Sherlock" papers that would be obvious from hindsight. The fact is, LLMs are a higher-order construct in machine learning, much like a fish is higher-order than a simple cellular colony. Low…

NLP/Computational Lingustics is NOT a new field. The ACL was founded in 1962 LM research is also old, papers using very shitty LMs (I.e Markov chains) and discussing them have existed since the early 2000s and likely before. Check yourself before you try to check others.

Parent post is talking about LLMs, i.e. Large LMs. Research on LLMs is indeed in its infancy.
Post reply on HN