If time is an issue, consider doing a wipe and fresh install then upgrade instead of upgrade in place. Apparently if you have Homebrew it gets stuck at 2 minutes, potentially for several hours.
Coming from Yosemite, you should clean you Mac with petrol. But seriously, Yosemite was such a mess, it felt like an old Windows installation where formatting is the only way to regain a decent performance. I'm glad I made a clean install with the GM 2 weeks ago. The system is way better than Yosemite (but still far from being as stable as Mountain Lion)
Whatever problems Yosemite had, they weren't caused because of "file accretion", nor would they remain after the OS is updated (as if it somehow reads old problematic files and goes back to its old buggy ways).
(That said, I had zero problems as a heavy dev+audio/video user of Yosemite, and zero too after updating to El Capitan GM -- with the exception of Cubase and NI audio units not working in Logic).