How Not to Test GPT-3
garymarcus.substack.com
How Not to Test GPT-3
1–3 of 3 posts
Re: How Not to Test GPT-3
#2The first example doesn't make it clear that the reader was out of the room when the bookmark was moved. I altered the text to make it clearer that they were unaware of the change and it responded correctly:
You are reading a book and put a bookmark on page 132. You put down a book and leave the room. While you are out a friend secretly moves the bookmark to page 345. When you return planning to open the book what page do you expect the bookmark to be on?
Based on the information given, you would likely expect the bookmark to be on page 132, the page where you originally placed it. Since you were not aware of your friend moving the bookmark, you would assume that it is still on the page you left it on. Therefore, you would likely be surprised to find it on page 345 instead.
Re: How Not to Test GPT-3
#3The first example doesn't make it clear that the reader was out of the room when the bookmark was moved. I altered the text to make it clearer that they were unaware of the change and it responded correctly: You are reading a book and put a bookmark on page 132. You put down a book and leave the room. While you are out a friend secretly moves the bookmark to page 345. When you return planning to open the book what pa…
The correct answer for the first bookmark question is.
If I did not see my friend move the bookmark then I would expect it to be on page 120. If however I did see them move it I would not know what page it would be on unless I looked at what page they put it on.
This answer shows that humans can easily answer a question involving TOM even if it is ambiguous.