I think about this a lot, too. My academic interests are pretty broad, and I could improve in every subject, so why don't I? I think there are two reasons: a lack of
focused effort, and the steadily increasing demands of adulthood.
I do reasonably okay at self-guided education when I want to, but there's definitely a difference vs. a structured secondary education environment, where there is accountability and other people to guide each other through the process. And, that's coming in to those subjects with already a better-than-average literacy and numeracy; I have to expect that for people who struggle with grade school reading comprehension or math, trying to bootstrap those abilities alone would be daunting.
Also, there's just less room for pursuing those now. Lots of people are getting squeezed by concerns that aren't part of most childrens' awareness -- housing costs, bureaucracy, the treadmill of maintaining all the machines that get us through daily life. Those add enormous pressure to dedicate more time towards professional development and "getting ahead", or at least not falling further behind, and that has been eroding all of the unstructured time that I would spend working my way through a textbook (or online class). People with poor literacy are probably more likely to have lower-paying positions, so all of those demands are even more severe.
Not that it's impossible. Lots of people do manage to self-educate their way out of poorer circumstances, and certainly the internet has made that far more accessible than it was before the turn of the century. But, let's not underestimate how challenging it is, either.