Earlier quoted context omitted.
Seriously? If your application become sufficiently complex that it can't be easily grokked by a decent dev, that's when you know it's time to switch. Until then it's just over engineering (which the JS community admittedly loves).
Very few teams have the opportunity to switch later on, and if they do, it's at the expense of the job of the person who chose to go without a framework.
I estimate that every 20% that a codebase grows it needs some amount of refactoring. Definitely anything that moves from skunkworks to production will need some pretty heavy rewrites, and will probably need a framework that better meets its newer needs. The question is how much cost was sunk in getting there. The refactor will have to happen no matter what.