In the same vein, I was playing around with its ability to annotate code with explanations of the changing constraints on different variables (I thought this was a good proxy for deeper understanding of code).
I asked it to write a moderately complex algorithm, and it chose to write a function to compute the Pollard's Rho algorithm.
Then, with a few example comments to show what I meant, I asked it to annotate the code to show how the constraints on the variable changed as each line was executed.
This was its output:
# n : an integer n > 3, no upper limit
# seed : an optional integer parameter (default = 1) for the polynomial function
# pollards_rho(n, seed) : an integer or nil, representing a non-trivial factor of n, or nil if the algorithm fails to find a factor
def pollards_rho(n, seed = 1)
return n if n
end
This is not actually the most impressive thing I've seen it do, but it illustrates that, even if GPT doesn't "really understand" code, it can emulate understanding to an impressive level. I'd go further and say that it opens up new questions as to what understanding actually means.
One personal "woah" moment was asking it to write some unit tests for a simple 2d game GPT and I wrote together. One function, "create_area" took a 2d array of characters (representing a map) and four integers representing coordinates, and a tile type. (The purpose being to create a rectangular area of the desired tile on the map according to the passed coordinates.)
GPT-4 successfully figured out how to write a unit test: it created a 5x5 array of ROCK tiles, passed it to create_area with the coordinates 1, 1 and 3, 3, and successfully figured out what the output should look like, even writing a fairly concise test to check the output (modified) 5x5 array. This was an eyebrow-raising moment for me: it made clear that GPT really does emulate some kind of "computation" internally, though quite possibly in some abstracted form. The geometric nature of this problem stuck out to me: a human can "see" a 2d array as a rectangular grid, and might realise the function carved out a smaller rectangle from that grid, but I never expected to see a computer (let alone a language model) figure it out. Interesting times, indeed.