Earlier quoted context omitted.
Well, we still have religious people doing pilgrimages. Both christians and not. At least some of them sure do treat them as vacation. On top of that, plenty of people in medieval time were not that deeply religious. Even in clergy or among monks. Even including religious component, I don't really see how it differs much from e.g. pilgrimaging to music festivals. I've done quite a bit of traveling where solo task was…
To someone religiously devoted it's not an impractical diversion but something deeply practical. It seems to me in some sense more akin to traveling to a renowned hospital -- you might take in some sights while you are there, but diversion isn't the point.
You might be conflating shightseeing and vacation as time-off. On top of that, you might want to look into how pilgrimage or other religious get togethers look like.
It's not like people come and pray and do nothing else. In most cases shopping and getting together with people from afar is involved. People have and always had natural needs like roof for the night and food. And rest during the day. If a bunch of people got together at night having a similar goal, they'll probably have a chat and maybe even party a little. There's a reason why many religious sights were surrounded by shopping and entertainment. Even a long time ago.
For example, the very simple act of sunday church. Back when traveling took time, it was certainly a day trip for rural people. It was pretty much a tiny conference or festival for them. That's why market square a frequently next to churches and markets are held on sunday. People come, shop, have a chat... And go to church while at it. I'm talking not only today, but from historical perspective too.
Talking about vacation only as "diversion", I guess quite a few of my vacations wouldn't count then. If I was going to a music festival, diversion itself wouldn't mean jackshit to me. I went to many festivals where I basically got to the location in cheapest possible way. Stayed on festival grounds and then got home in cheapest possible way. I saw no sights nor diversion itself was the point. But boy if it was vacation :)
Same about amateur racing. I get to the location, I race, get some food, sleep it off and go home. Sometimes terrain/location is part of signing up for the specific race. Sometimes I go there just because it's well organised and my friends are going. More likely than not no sights are seen nor I are about location aside from race track much.
Sometimes vacation is purely to get out of daily routine and jut spend time with myself. Diversion itself ain't the point either.