Earlier quoted context omitted.
Again, the only way to tell if overfitting specifically (and not other factors that are more likely) is the issue is performance on a held out test set.
That's the only empirical way, yes. We can also just talk about what it would mean in theory, though. In this case, we'd say that AlphaGo is well trained to the training data set sampling distribution but that may be far from the actual world game distribution.
That does not relate to overfitting. That problem is closer to "domain adaptation."