Those people who don't go to college just go get a job instead. (If jobs need training, they can apprentice or otherwise learn-on-the-job, but they won't be spending half that time learning french literature or what-have-you) Since people are doing something more productive with their time than sit in a classroom, their lifetime productivity improves, hence the productivity of the economy as a whole improves.

Given that college is a bad investment for most people, we'd be better off if fewer people did that. (Though the optimum is probably that some people go to college, not all or none.)