I'm in my 40s and have been designing and building software for money since the early 90s (and coding since the 70s). I remember asking these same questions at some point, while watching older developers lose their mojo (often ending up in management). I worried that I would end up like those soulless managers and burnt out architects. I didn't. Not yet, at least.
> Do you have to go into management to continue progressing upwards in pay and influence? I know this isn't the case at some companies (e.g. Google), but is it rare or common to progress as an individual contributor?
You will always have more influence as a VP, Director, or general board member. Architects and team leads can be part of the management group, but actively avoiding or despising it is alienating those who carry financial responsibility for the company. Once you have the ability to make long term and rational architectural decisions, you will want to be able to use that knowledge to change things. Making things happen outside of the management structure requires a great deal more force than from within.
But, you can retain your technical edge while in management (at many companies at least). I am an architect and CTO at one company and a board member at two other companies. I also code almost every day, as I believe that software design and architecture cannot exist without understanding how things work today.
That said, you don't have to go the management route. I do suggest at least making peace with management and managing, as it is a valuable tool for getting things done when mad coding and design skill is not enough.
As for pay ...
> Is there a plateau in pay? Is there a drop in pay switching jobs after a certain number of years experience because places are looking for 5+ instead of 20+?
Yes, pay rates tend to plateau if you're not part of the management or directorship. There are exceptions to this, including a number of smarter employers or if you change jobs regularly. You can also start your own company, but that requires both management and business savvy, and adds some risk.
I have only changed employers a few times as I've been lucky to really enjoy my teams, but I do own a consultancy as well (which allows me to adjust for any ceiling at my day job).
> Are older devs not looking for new jobs because they have families and want more stability/are focussed elsewhere?
Many older developers end up in management, owning companies, or as architects (who gravitate toward larger, older companies). Most older devs prefer stability, but not all.
> Is becoming a specialist rather than a generalist the answer?
The answer to what? Specializing will allow you to do more of something you want to do. Generalists often do better with entrepreneurship and general opportunity. You want to make more money? Management and ownership are great routes for that, and generalists excel in those roles (in my experience).
> And lastly: if you're in your late 30s, 40s, 50s, what are you doing at your job? What are the older people in your workplace doing?
I spend about half of my time designing systems and interfaces (from APIs to UIs). I spend half my time prototyping and setting up projects for my teams. I spend the last half of my time making sure it gets done properly. I still have more ideas for products than time, and I still pick up several new tools a year. I'm always learning, and always improving my own methodologies (as well as my team's). I still love what I do.
I also work with a software architect who is in his late 60s who is still both passionate and coding daily. He avoided management and does not often regret it, and has coded everything from OS subsystems (in the 60s) to iOS and web things today. His rate of learning has slowed down appreciably, but his vast knowledge and experience more than makes up for it. He was the first architect I met that still loved what he did (when he was in his early 50s).