He was an incredibly great man, and it remains a privilege to have learned from him. While I left mathematics for engineering, his audacious asking of the right questions around the philosophical foundations of an endeavor remains a large influence on me and has become a hallmark of my engineering work. It makes me smile that you are familiar with him as well. :)
---
Here's the full puzzle, as best I remember it:
Suppose you have two convex polygons with the following very specific relationship: one has been created from the other by making one side stretchy, moving an adjacent side on a hinge, and keeping the remaining sides fixed. For example, imagine a square with a top side made of rubber, and a rigid right side hinged at the lower right corner. You can make a series of convex polygons in a continuous fashion by rotating that right side on its hinge.
The question is, if you have two convex polygons that differ only by this one stretchy side and this adjacent hinged side, can you guarantee that you can always smoothly deform the one into the other by this hinging method while keeping the whole thing convex?
That is to say, if you are deforming one polygon into another by this hinge and rubber band method, if your starting polygon and ending polygon are both convex, are all the middle ploygons guaranteed to be?
The answer is intuitively obviously yes, but in point of fact, it is no.
---
To bring this back to the original story, the question was a small step in a larger constructive proof he was working on. The overall result was already known - in fact, we had just discussed it in class - but the proof had this distressingly jerky, discrete movement to it, and he was hoping to construct a more pleasing and smooth algorithm as a more satisfying proof.
As for me, I would not have known how to begin to prove even the smaller question... but I sure could doodle a counterexample. ;) I therefore looked for a one with all the gusto of a young grad student hoping against all odds to do something helpful. You may look with all the confidence of knowing there is something to find, which is also a tremendous help.
I only know his side of the story because he started Monday's class with this line: "I spent the entire weekend trying to prove the result, without success, and it was a good thing too, as there was a counterexample in my box this morning." He did seem genuinely frustrated, but I also wouldn't have put it past him to have exaggerated that part for the laugh.
Anyway. Asking an AI to find counterexamples under such circumstances seems to me similarly reasonable to asking grad students. In my engineering work, I find there is a balance between using the AI to improve and augment your work (especially to call on the diverse perspectives in its training set), and using the AI to avoid your work. I do think the best experience and results are found in that balance. I would expect the same to be true in mathematics.