I think it's important to accept that there is a group of people - perhaps small - that love reading literary fiction. It's not reasonable to assume bad faith, just because you can't see the merit in something. There maybe people that use it as a means of identifying with a group and not out of genuine passion or interest - but that is true of every group. You always have people who attach themselves in this way: programming and tech world are a classic case.
Anyway just because something seems esoteric does not mean the person iis not enjoying it. We all have radically different needs, experiences and interests. And it's OK. Anyone who thinks that because they read literary fiction they are superior in some way is foolish certainly. On the other hand they may quite reasonably say that they consider one form of fiction to be aesthetically superior: that is the nature of aesthetics - we all get our say.