I think it's a lot easier to build "competence" than people usually think.
Think back to college, to your friends who actually got degrees in various specialties. How many of them actually closely read even five of the books in the curriculum, took detailed notes, and thoroughly understood the material?
A motivated autodidact can do that in a couple months (about how long I manage to stay interested in any one topic). Understanding that 20% of the knowledge in the field really does tend to make you 80% "competent." I find that I can have "competent amateur"-level conversations with specialists in a subject after just that much effort.
From there, it's all a question of whether you want to move toward expertise. Becoming an expert takes MUCH more time, energy, and effort.
What normally happens for me is that, after reaching the level of competence on a broad subject, I may come across a subspecialty or two that inspire me to reach competence on them. That's the layer deeper.
That doesn't usually happen right after the broad review. It usually happens a few cycles later, but because I engaged the material and made notes on it, I can refresh what I knew from my overview and be prepared to dive deeper.
It opens up interesting conversations (I know something about an incredible variety of things, so can usually engage someone in a conversation about something that matters to them). It allows me to connect ideas that make me seem "creative" in any one particular field.
In other words it's a very engaging, fun way to live! And not nearly as daunting a thing as it seems if you mistake building expertise with building competence.
Being a polymath is just really good fun if you let it be!