This article crystalizes a point that was vague in my mind: "megacities have a problem making it affordable to raise large or even medium-size families. How many of us could live in San Francisco, London or New York City and bring up three children in a decent neighborhood with good schools? Only the already well-to-do can manage such a feat.". As a mid-30s person with a new baby, I don't see how the numbers work for…
I'm not really sure how moving to Vancouver helps solve any of your problems. If you're priced out of Milton you're likely priced out of the entire Metro Vancouver area.
Is rent in Vancouver that much cheaper than the GTA that you're saving money and it's worth moving? I don't think it is.