Live data from Hacker News

GPT-3 vs. 3M free-text trivia responses

watercoolertrivia.com

31–40 of 45 posts

Re: GPT-3 vs. 3M free-text trivia responses

#31
I think than human communication is better than any ai model because human goes from general to concrete, adding more information to convert general context into insightful ideas. We use language to communicate what is important for us. Ai is about navigating in an ocean trying not to be captured by concrete things, because it knows nothing about being real.

As a mathematical methafor, each word is a window in the tangent space of an n-dimensional sphere and the normal points to the center of the sphere, as time goes on communication is about shrinking the radious of the sphere so that the center is like the kernel of what we wish to communicate. The sphere in an ai system has a very big radius and is not able to shrink because it lacks a kernel.

You could simulate a human conversation with an ai, for example to sell something following steps like description of a product, price advantages and so on, but in a real context those are ad hoc methods and there is no general rule to shrink that is to penetrate deep into the meaning of what you want to communicate.

Re: GPT-3 vs. 3M free-text trivia responses

#32
post #8

"Two'fer Goofer", "Tough Training" - why would GPT-3 give the question back as its response in these cases?

GPT-3 has a habit of repeating its input. GPT-2 had the same problem, as did all language models before. LSTMs were really bad in this respect and would often get stuck in single word loops. GPT-3 just does it less often, and tends to repeat sentences or ideas, rather than single words. A larger corpus and more training seemed to have helped reduce the issue, and I think they tweaked the temperature algorithms as wel…

Isn't it also that in human writing (and training) the most common or correct response/followup to a prompt structured as a question often is a complete "x is y" sentence? GPT-3 appears to consistently parse these question prompts as requiring it to include certain strongly associated words in its output, but not consistently as trivia questions implying terse, associated-word-only answers

It's the opposite of the 'know all' the blog describes, it's GPT-3 knowing where to find the answers without even understanding it's a quiz!

Re: GPT-3 vs. 3M free-text trivia responses

#33
post #8

"Two'fer Goofer", "Tough Training" - why would GPT-3 give the question back as its response in these cases?

GPT-3 has a habit of repeating its input. GPT-2 had the same problem, as did all language models before. LSTMs were really bad in this respect and would often get stuck in single word loops. GPT-3 just does it less often, and tends to repeat sentences or ideas, rather than single words. A larger corpus and more training seemed to have helped reduce the issue, and I think they tweaked the temperature algorithms as wel…

We don’t have to imagine GPT3 doesn’t understand maths. It can reliably answer 1+1= because there are so many instances of it in the corpus, but if you give anything more complicated it will just throw out a random, incorrect, answer in most cases.

Re: GPT-3 vs. 3M free-text trivia responses

#34
> Less than 10 years later, a general-purpose open-sourced technology without the massive mainframe or cooling fans can compete on the same level.

First, are GPT-3 models generally available?

Secondly, GPT-3 is many orders of magnitude larger than any model existing 10 years ago. There are certainly cooling fans involved.

Re: GPT-3 vs. 3M free-text trivia responses

#35

Somewhat related, but I can't stop thinking about what's going to happen when GPT-4 reads articles like this. Articles like these that discuss GPT-3's output, or comments and articles directly generated by GPT-3, are going to be fed into GPT-4 as part of a theoretical WebText3 dataset. Will that help or hurt GPT-4's learning? What effect will it have? Theoretically you can feed GPT-3's responses back into itself and…

One intuition about the large GPT-models is that they are learning "facts" like they are learning grammar rules. It's just that their capacity is so immensely large as to learn facts as part of the language model.

Re: GPT-3 vs. 3M free-text trivia responses

#36

How is GPT3 considered publicly available? I applied months ago and have received no response.

I definitely agree it's not publicly available. I too applied in Dec 2020 and just the other day (09 March) received a key. Hang in there, it'll come :)

Applied in June and haven't heard anything!

Re: GPT-3 vs. 3M free-text trivia responses

#37
It would be helpful to see the prompt structure used for gpt-3: state of the art prompting methods do significantly better than naive ones, and randomness and other settings can make a big difference in quality as well.

There are also cyclical prompt methods which can help derandomize gpt3, for example, you could generate three to five answers per question then feed the set back to gpt-3 and ask it which is the most correct; the first round would use high randomness settings the last very low.

My own experience is that you sort of have to work with gpt3 to get on the same page sometimes, and when you do the results can be remarkable.

Anyway, fun idea!

Re: GPT-3 vs. 3M free-text trivia responses

#38
There was recently a thread[1] about an open-source alternative to GPT-3 here, called "GPT-Neo".

I hope it takes off. But the language on their website undermines some confidence: "GPT-Neo is the code name for a series of transformer-based language models [...] we plan to train and open source."

PS: The original URL that was submitted to HN is now 404; it moved to here[2].

[1] https://news.ycombinator.com/item?id=25819803

[2] https://www.eleuther.ai/projects/gpt-neo/

Re: GPT-3 vs. 3M free-text trivia responses

#39

Somewhat related, but I can't stop thinking about what's going to happen when GPT-4 reads articles like this. Articles like these that discuss GPT-3's output, or comments and articles directly generated by GPT-3, are going to be fed into GPT-4 as part of a theoretical WebText3 dataset. Will that help or hurt GPT-4's learning? What effect will it have? Theoretically you can feed GPT-3's responses back into itself and…

Your ascribing way too much intelligence to a very simplistic text prediction algorithm. Neither GPT-3 nor GPT-4 (assuming it will just be a bigger GPT-3) have any concept of identity that could influence their predictions in the way you seem to assume. If GPT-4 would be trained on text produced by GPT-3 and on text that analyzes that, what GPT-4 would learn is how to sound like GPT-3 and like detractors or admirers…

I mostly agree with you, but I wonder what's the distance between what you described and our actual intelligence.

Re: GPT-3 vs. 3M free-text trivia responses

#40

> The more important takeaway: dozens of technologists at IBM spent more than three years and untold millions of dollars building the program specifically trained for Jeopardy! prowess. Less than 10 years later, a general-purpose open-sourced technology without the massive mainframe or cooling fans can compete on the same level. This statement from the article confused me. GPT-3 is general-purpose, but not open sourc…

Yep, it's not only open sourced, it's not even open access. (I get the reason, but somehow, maybe due to the fact that it's from an organization called OpenAI, a lot of people seem to think that everyone could use it if they wanted to.)
Post reply on HN