If you go to an Ivy, everything you wrote is incredibly true. Going to any Ivy or other elite (or even semi-elite) school is a HUGE benefit if for no reason other than networking.
As you move down the scale that becomes less and less true. People go, learn whatever they learn, get few if any connections, and wind up with a pile of debt.
So, we've created an incredibly regressive system.
One other thing to note. When people form those friend groups it's not just about who's their intellectual peer, it's also about who came from similar backgrounds etc. It's an engine of division as much as anything else.
I agree that the value is there for some individuals, but I disagree that this is a good thing for society as a whole. If you get accepted to Harvard, it's almost certainly worth your while to go. College still remains a finishing school for the elites in our society. Some people bubble up from the underrepresented portions of our society, but they remain the exception, and will continue to be so until something changes.
I've included a couple links below that explain the disadvantages some groups face when attending college.
http://www.pewresearch.org/fact-tank/2016/07/27/blacks-with-...
https://www.vox.com/2017/9/11/16270316/college-mobility-cult...