You could, there are many approaches to design interpretable/explainable systems (Here
https://distill.pub/2018/building-blocks/ is a nice and recent review of some of these problems with illustrations), but there are serious tradeoffs so people usually choose not to do so.
The first issue is that, in general, explaining a decision is a harder task than making a decision, just as reviewing and understanding someone's decision takes more effort than it took to make it. This means that adding explainability and interpretability to a system that achieves some result will take more effort, time and money than building that system itself - you'll triple your budget and get no better actual results, just a better feeling about them. This also limits the effort invested in tools and methods for this; there's large demand and funding for making stuff work or making stuff a bit better, so many people are getting paid to work on that, and not nearly so much on interpretability.
The second issue is that best performing systems take "everything" into account and thus become too complex for a human to understand. A system can be explainable, interpretable and accessible, but if making a correct decision inherently relies on complicated interrelations between many factors, then it will be difficult (or at least time-consuming) for humans to understand anyway. If you limit the complexity of the system to something that can be easily explained, then you're severely limiting its power, and you get poor results.
For your query planner, the analogy is the scenario where we could build a query planner that is capable of optimizing beyond the commonly used basic blocks in query plans, instead outputting optimized machine code to perform particular query better than the standard blocks, achieving improved performance by merging and interleaving operations. The resulting output would be much harder to interpret, since no "block-level" query plan can be equivalent; and requiring the operations to be neatly separable in a limited number of blocks each with separate human-understandable meaning would mean restricting the system and inevitably getting a worse result.
The same applies to humans, by the way - being able to explain all the factors why you believe your decision is likely to be correct takes more knowledge, insight and effort than simply making that decision. It's not like humans can explain why exactly they see (or don't see) an animal in picture like http://www.naute.com/images/cow.gif , consciously understanding that in a structured way is very, very hard. If you would require a human to explain the criteria they'll use before they get to make a decision (as opposed to post-decision rationalization that often has no relation to why the decision was made, and has more relation to making up a socially acceptable plausible justification why you did what you did), then you get poor, simplified criteria that can't reflect the complexity of the problem; you could get their "rules of thumb" or their major factors, but explaining why exactly it is so is generally a mentoring task that takes months (if not more) of effort both from the teacher and the student.
AlphaZero system plays Go and Chess better than any human. It could be modified to describe in detail why it chose this move over the other, but would you (or any other human) have the mental capacity to truly understand it, no matter how well it's explained? The distill.pub article illustrates lots of ways how an image classifier can explain why it made one choice or another - but is it actually useful for tasks other than for R&D to debug or improve the model? It seems like a nice-to-have feature that would be liked if it's not to expensive, but I'd bet that for most models the company wouldn't actually use the explainability feature (again, outside of its builders debugging or improving it) because the whole point of ML is that people get removed out of the loop and don't look at the decisions.
