Earlier quoted context omitted.
This is a good point, and I expect in the near future we will be able to simply ask a neural network to explain and change its behavior without retraining. I already see signs of that in prompt engineering used to interact with GPT3 or Dalle2.
It's just going to be as good as asking a human to explain its behaviour. It might be a semi-accurate interpretation of its actions based on its internal knowledge, but it's never going to be the actual thing. The actual decision making inside a neural network is fundamentally not something you could simplify into language exactly.
Even if the given reason is simple, like "I decided to slow down because the car in front of me is red", explicit override of the learned rule ("don't slow down when you see red cars") might potentially increase the chance of the accident by a lot more than 0.0002% because we are messing with the model decision making process.