Earlier quoted context omitted.
I'm no mathematician but it seems like if this happens, we get to a quite intriguing place as a species. Say we achieve interstellar travel, but nobody actually knows how it works. Or we cure cancer, but the "cure" requires a microrobotic implant, and it runs as a blackbox AI, and only the other AIs can make one, and there's no guarantee they will know how to make one tomorrow. Or we solve global warming but it requi…
I think anything is and will be explainable. Like in the OpenAI proof, I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. After all, the point of science is so that knowledge can be made logically transparent. If something can’t be explained, it isn’t really understood yet — and the same applies to model outputs. The only question is how m…
I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus?
The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it?
I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.