GPT-3 vs. 3M free-text trivia responses
watercoolertrivia.com
GPT-3 vs. 3M free-text trivia responses
1–10 of 45 posts
Re: GPT-3 vs. 3M free-text trivia responses
#2> Credit where due: friend of WCT Dennis wrote the script to feed Water Cooler Trivia questions to GPT-3 and access the OpenAI API.
It should be noted it's against OpenAI's rules to share access to GPT-3, although they've been inconsistent about enforcing it.
Re: GPT-3 vs. 3M free-text trivia responses
#3> Credit where due: friend of WCT Dennis wrote the script to feed Water Cooler Trivia questions to GPT-3 and access the OpenAI API. It should be noted it's against OpenAI's rules to share access to GPT-3, although they've been inconsistent about enforcing it.
We gave him a spreadsheet of questions and he gave us a spreadsheet of GPT-3 responses, so probably not actually shared access :)
But point well-taken!
Re: GPT-3 vs. 3M free-text trivia responses
#4> Credit where due: friend of WCT Dennis wrote the script to feed Water Cooler Trivia questions to GPT-3 and access the OpenAI API. It should be noted it's against OpenAI's rules to share access to GPT-3, although they've been inconsistent about enforcing it.
We gave him a spreadsheet of questions and he gave us a spreadsheet of GPT-3 responses, so probably not actually shared access :) But point well-taken!
And yah, I hope I didn’t violate their terms by doing this —- it seemed like a fun thing to do!
Re: GPT-3 vs. 3M free-text trivia responses
#5> 2. Clues confuse GPT-3.
They should probably have been removed. This gives me the overall impression that the testers treat GPT-3 a bit too much as something like an artificial human, and not enough like an algorithm (which will work better with sanitized input). This is not a major criticism, the experiment is still interesting.
Could it be that the marketing from OpenAI it to blame? From the OpenAI front page:
> Discovering and enacting the path to safe artificial general intelligence.
> Our first-of-its-kind API can be applied to any language task, and currently serves millions of production requests each day.
Does that seem misleading?
Re: GPT-3 vs. 3M free-text trivia responses
#6> 2. Clues confuse GPT-3. They should probably have been removed. This gives me the overall impression that the testers treat GPT-3 a bit too much as something like an artificial human, and not enough like an algorithm (which will work better with sanitized input). This is not a major criticism, the experiment is still interesting. Could it be that the marketing from OpenAI it to blame? From the OpenAI front page: >…
I thought about removing those two-word clues but ultimately, they exist for users so I wanted to be closer to apples-to-apples!
Re: GPT-3 vs. 3M free-text trivia responses
#7> 2. Clues confuse GPT-3. They should probably have been removed. This gives me the overall impression that the testers treat GPT-3 a bit too much as something like an artificial human, and not enough like an algorithm (which will work better with sanitized input). This is not a major criticism, the experiment is still interesting. Could it be that the marketing from OpenAI it to blame? From the OpenAI front page: >…
OpenAI's marketing has been fair, and it's not misleading (GPT-3 can be applied to any language tasks but your mileage will vary as this submission demonstrates).
However, OpenAI's endorsement of the demos amplifying and anthropomorphizing GPT-3 as a sentient mind don't help, and it's been disappointing that OpenAI doesn't really push back on that. (the hype is what prompted my rebuttal on GPT-3 expectations, which still holds up: https://news.ycombinator.com/item?id=23891226)
I do believe the future of AI text generation is more bespoke, algorithmic-input-friendly models which is what I've been working on as my side project.
Re: GPT-3 vs. 3M free-text trivia responses
#8"Two'fer Goofer", "Tough Training" - why would GPT-3 give the question back as its response in these cases?
Re: GPT-3 vs. 3M free-text trivia responses
#9How is GPT3 considered publicly available? I applied months ago and have received no response.
Re: GPT-3 vs. 3M free-text trivia responses
#10Somewhat related, but I can't stop thinking about what's going to happen when GPT-4 reads articles like this.
Articles like these that discuss GPT-3's output, or comments and articles directly generated by GPT-3, are going to be fed into GPT-4 as part of a theoretical WebText3 dataset. Will that help or hurt GPT-4's learning? What effect will it have?
Theoretically you can feed GPT-3's responses back into itself and ask it to introspect about them; whether it was right or wrong and to comment on why it gave the response it did. But I doubt GPT-3 is particularly good at self introspection. GPT-3 was trained before GPT-3 existed, obviously. So it was never trained on articles analyzing the output of an AI of its caliber.
But GPT-4 is going to be trained on a corpus filled with people analyzing GPT-3's outputs, like this article. We would expect GPT-4 to be able to write an article like this. So it should be theoretically possible to give GPT-4 its own output, and then ask it to provide introspection, and for that introspection to be insightful.
EDIT: Follow up thought. It's almost as if the internet is being filled with a training corpus on GPT-3's failings. Every fact that GPT-3 failed to learn from WebText2 is now going to be repeated, alongside the correct answer, in WebText3. Humans are globally working, unknowingly, to build a curated dataset by which GPT-4 can learn from GPT-3's mistakes.