Earlier quoted context omitted.
Why would the simplest theory be the most likely to be true?
Less edge cases mean it's less likely to break if we discover a new particle or something else unexpected
I guess that's a good overview. It's related to the phenomenon of overfitting in machine learning: you can always easily find a sufficiently complex (or complicated, large if you prefer) theory fitting all data points. Because this theory simply encodes each observed case (including progressive sophistications of encoding), you naturally expect it to fail on unobserved cases -- it makes no effort at generalization. The simpler theories have a greater chance of generalization, they're more likely to be the "true" mechanism of the process that doesn't simply encode "edge cases" as you cited, and thus are more likely to also work on unobserved data.
Honestly I haven't seen attempts at making this process more rigorous, when applied to physics. There's the large corpus of machine learning study which provides concrete results and even concrete comparison tools, but the times I've asked a physicist I've been dismissed; while it seems incredibly valuable in the face of lack or large cost of experimental data, which is quite relevant today.