> how did they survive in the role for any length of time?
Not all knights saw that much action. I agree though that they had strong incentives, besides survival in battle, to be at least defensibly competent.
As to those incentives -- they had a multi-generation arrangement with the sovereign: to supply a certain number of mounted men when he called for it, maybe determined by the size of landlord they were, and in return the king let them keep dominating economic activity in their usually-rural area. The knight held the purse strings and could extract as much as he felt prudent from his tenants (farmers etc).
But the knight would employ Men-at-Arms. These were the guys with what we think of today as 'Knight Skills,' but no land/title/purse. They'd be taken care of, and they'd definitely try to keep their boss alive. Or sometimes an older knight might send the required men without going himself. (This varies a lot by time period of course).
So on the one hand, they didn't have to do superhero stuff if they didn't want to.
On the other hand -- their whole reason to exist was intensely martial; the structure of society reflected the fact that folks riding horse-tanks dominated all of civilization. There were good reasons for someone with a coveted horse-tank position to, yes, be able to be ride, be at least minimally competent, and show up to risk their lives when the king calls for it, or else risk losing their privileged position.
> the reason anyone bothered to work for them at all, rather than towns responding to recruitment attempts exactly the same as invasion attempts, with stoning, boiling oil from the rooftops, etc
Well, the knight owned the siege-proof fortresses in the area; that's one of those things they did well in feudal times, picking geographical strong points for fortresses.
The knight also owned the town people lived in too, the farms, houses, etc. His family got them way back when kings started deciding that they didn't want the town to send 5 random foot soldiers when he called -- he'd rather have 1 professional, well-equipped horseman with mail, shield, lance etc, and if you could make that happen he'd just give you your neighbor's stuff to keep making it happen in perpetuity.