I think the problem might be in the training data. Famous code examples are probably copied a lot and therefore appear multiple times in the training data, prompting the neural network to memorise it completely.
Famous code examples are also much more likely to be noticed. For all I know, the thing might be spewing random GPL'd code from the long tail of GitHub all the time and nobody notices because it was written by some random guy and not John Carmack.
Well, it's sure speculation on my part what the root cause is, but i think OpenAI is already trying to ensure the network generalises. It's just common behaviour for neural network to memorise frequent samples, so I think my guess is quite realistic. I don't think OpenAI would not notice large-scale memorisation in their model. But as long as they don't publish more details it's just guesswork.
Just keep in mind that it's a statistical tool. You can't really formally prove that it won't memorise, but I think with enough work you can get it unlikely enough that it won't matter. It's their first iteration.