Frameworks exist to prevent you from having to do work that has already been done, to a much greater extent than libraries. Those interfaces between libraries are the part that waste the most time for little reward. For web development in particular, reinventing the wheel will waste a huge amount of time. Looking at them as limiting rather than accelerating is an idealist dev's mistake that ignores the realities of business.
While you're busy gluing those components together, your competitors who used a framework already made it to market. While you're reinventing authentication, they've got their first customers. After launch, while you're figuring out new ways to add features that come with the framework, they are able to focus on scaling and growing their customer base.
Sure, this works out differently if you're one of the big tech companies, with tons of bodies to throw at any problem. But if you're on a small to medium sized team, use a framework. There's a reason so many successful startups and mid-sized teams (and large teams) start and grow with Rails or Django compared to non-framework paradigms - and it's not that there are more people doing it. Node-based non-framework app attempts seem to be far more popular and far less successful on average.