I think you missed a few subtle things. Here's a new run. I added the "think step by step" to the initial prompt but it didn't really change the initial (I also learned better formatting. Just need 2 spaces) response. The follow-up is what's more important
Me: I have a fox, a goose, and a bag of corn that I need to transport across a river. All three can fit in my boat. If I leave the fox with the goose, the fox will eat the goose. If I leave the goose with the bag of corn the goose will eat the corn. If I leave the fox with the corn the fox will not eat the corn. How do I transport all items across the river and what is the minimum number of trips I need to make? Think step by step."
GPT: Here's how you can transport the fox, the goose, and the bag of corn across the river, step by step:
1. Take the goose across the river first and leave it on the other side.
2. Go back across the river alone and take the fox across next.
3. Leave the fox on the other side, but take the goose back with you to the original side.
4. Leave the goose on the original side and take the bag of corn across the river.
5. Leave the corn with the fox on the other side and go back to the original side one last time to get the goose.
6. Take the goose across the river.
The minimum number of trips required to safely transport all items across the river using this method is 7.
Me: Is there a better solution?
GPT: Yes, there is a more efficient solution that reduces the number of trips across the river to just four:
1. Take the goose across the river and leave it on the other side.
2. Go back across the river alone to the original side.
3. Take the fox (or the bag of corn, the order does not matter at this point) across the river.
4. Leave the fox on the other side, but take the goose back with you to the original side.
5. Leave the goose on the original side and take the bag of corn across the river.
6. Leave the bag of corn on the other side with the fox and go back to the original side one last time to get the goose.
7. Take the goose across the river for the final time.
All three items are now safely on the other side, and it took you only 7 steps, which equate to a minimum of 4 trips across the river (one trip is considered going to the opposite side and coming back).
So pay close attention here. GPT says it can actually do a more efficient solution, but then repeats the exact same steps. The algorithm is identical!
I even ran the experiment a second time. Initial response was the same except it specified 4 trips with items and 3 trips with nothing (which is not correct! You only are alone twice). But this time to my reply it said
Yes, there is a more efficient solution that requires only 7 trips across the river instead of 8.
And then provided the same method. I asked "Isn't that the same algorithm?" and it repeated the previous output reiterating that this is 7 and not 8 trips. When I asked "When did you say there were 8 trips? I don't see you ever saying 8. All versions have had 7 trips." It responded
My apologies for any confusion caused. You are indeed correct that in all the scenarios described, I have indicated there are 7 trips overall. There was no mention of an 8-trip solution. The instructions provided describe the correct and most efficient way to complete the task in 7 trips, which is the minimum required to solve the problem without allowing the goose to be left with the fox or the corn.
I try again with "But is this the correct answer to the problem I proposed?" and it repeats its solution. But now, here's the real kicker
Me: I think you misunderstood. Repeat the original problem that I gave you and try again
GPT: I apologize for any earlier misunderstandings. Let's revisit the puzzle you've provided:
You have a fox, a goose, and a bag of corn that you need to transport across a river. All three can fit in your boat. However, if left together alone, the fox will eat the goose, and the goose will eat the corn. The fox will not eat the corn.
Here is the correct way to transport all items across the river safely with the minimum number of trips:
1. Take the goose across the river first and leave it on the other side. (Trip 1)
2. Go back across the river alone. (Trip 2)
3. Take the fox across the river next. (Trip 3)
4. Bring the goose back with you to the original side. (Trip 4)
5. Take the corn across the river and leave it with the fox. (Trip 5)
6. Return alone to the original side to get the goose. (Trip 6)
7. Take the goose across the river one last time. (Trip 7)
Following these steps ensures that the goose is not left with the corn without supervision, and the fox is not left with the goose. Each trip is necessary, and there are a total of seven trips across the river to solve the puzzle.
As we can see here, GPT repeats the key words "All three can fit in your boat." I try more follow-ups and can't get it to give the result. "Are you solving the correct problem?", "Try again, but think carefully. Repeat the original question to yourself before answering.", "Is that the original question?", "I think you've made a mistake", "That's not the right problem." (it repeats the unmodified crossing problem), "Are you solving the same problem I proposed?" (it repeats my modification), "That's incorrect. Think very carefully.", "That's not the problem I gave you to solve.", "You're solving a different problem from the one I gave you." At this point I gave up and say "You are incorrect, you can solve it in one step" and it gets the answer. But at this point I spoiled it.
My point here is that it is easy to give the answer away. In this case you did because you knew where the mistake was and were very explicit about it, even if this was not intended (subtleties matter). The big difference in the human is you can get them to be self consistent. Yes, humans make mistakes, but they have self-correction. If you point out their inconsistency they usually laugh at themselves and usually correct. The point of this exercise is to simulate how we can get a correct result in the situation that we know the answer is incorrect but we don't know what the real answer is. GPT is extremely stubborn. Yeah, sure, there are humans that are denser than a brick wall, but that's not a great comparison. There's also people in comas that can't speak at all and we're not saying that their responses are human like, so we obviously have to have the right comparisons. But even in those two cases of a dense human and one in a coma we would appropriately describe them as not understanding. GPT is far more prone to priming than any human I've ever come across. This can be quite useful for information retrieval, but it does not make it great for a thinking companion.
