ChatGPT-4o vs. Math

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error.

I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'.

It figured it out and corrected it:

    Let's re-examine the image and the data provided:

    * The inner radius r1 is given as 5cm

    * The outer radius r2 is given as 10cm

    * However, the dimensions labeled "5 cm" and "10 cm" are actually the diameters 
    of the inner and outer circles, respectively, not the radii.

Then recomputed and got the right answer. I asked it if it could surmise why it got the wrong answer and it said, among a number of things, that math problems commonly operate in radii instead of diameter.

I restarted with a slightly modified prompt:

    There is a roll of tape with dimensions specified in the picture. 
    The tape is 100 meters long when unrolled. How thick is the tape?

    Examine the image carefully and ensure that you fully understand how it is labeled.  
    Make no assumptions.  Then when calculating, take a deep breath and work on this problem step-by-step.

It got it the first try, and I'm not interested enough to try it a bunch of times to see if that's statistically significant :)

Reminds me when I used to ace Ancient Greek translation tests (it's a thing in Italy) by looking up whole translated sentences listed as usage examples in the dictionary

I labeled it choke because it just stopped.

Similar to the article, I haven't found complementary image data to be that useful. If the information is really missing without the image, then the image is useful. But if the basic information is all available textually (including things like the code that produces a diagram) then the image doesn't seem to add much except perhaps some chaos/unpredictability.

But reading this I do have a thought: chain of thought, or guided thinking processes, really do help. I haven't been explicit in doing that for the image itself.

For a problem like this I can imagine instructions like:

"The attached image describes the problem. Begin by extracting any relevant information from the image, such as measurements, the names of angles or sides, etc. Then determine how these relate to each other and the problem statement."

Maybe there's more, or cases where I want it to do more "collection" before it does "determination". In some sense that's what chain-of-thought does: tell the model not to come to a conclusion before it's analyzed information. And perhaps go further: don't analyze until you've collected the information. Not unlike how we'd tell a student to attack a problem.

sometimes for physics/math exams, we'd get to create our own 1 pager cheat sheet to use. I'd just cram tons of actual problems/solutions on there, then scan for similarity.