Earlier quoted context omitted.
It's not a class thing. Fox hunting, horse riding, polo and skiing... yes. Conkers, No.
Wherever there is a chestnut tree you'll get conkers. I would imagine it's more of a town/village vs city thing rather than class thing.
I'm sure there are parts of the country where they're less common, but there's huge numbers of conkers falling off trees in big British cities (even if the majority will be in parks) as well as in the countryside. We played with them at my pre-teen city centre school for sure, and the trees are a common sight on roads and in gardens as well as public parks.
edit: the Woodland trust actually says "Though rarely found in woodland, it is a common sight in parks, gardens, streets and on village greens."