My experience is that some people genuinely are less productive without the social pressure net of their co-workers around them, or literally the boss on-hand as a sort of "productivity enforcer." It's varied wildly from job to job, office culture to culture, and at an individual level, but I take all those experiences with me and it tints my perceptions of productivity as well.
I remember when I was a recruiter it was a life of KPIs handed on down from on high. Management would pat you on the shoulder as they walked by if you were on a call, expressing approval of your activity. We'd crack jokes only somewhat ironically like "any time is a good time for a client call!" If you were still in the office at 7, you could expect a heartfelt appreciative email from management. If you hadn't been on the phone for an hour, a boss may shout across the room "bit quiet over there, someone check if komali_2's napping!"
Then on the days that all of management was at an off-site, things got a lot "looser." Less calls, less grandstanding, more hanging out in the kitchen, longer lunch etc. I don't know if productivity was actually lower ( it probably wasn't ) but it felt that way, because of the culture of our work place.
To that business, the idea of working from home would be laughable. Never mind that there were independent recruiters working out of their house and pulling in literally millions in revenue. How could you be as productive as an office of hardcore recruiters with their managers setting excellent KPIs and tracking goals and establishing milestones and shouting "a bit quiet over there!" occasionally?
Of course now that I'm an engineer I'm easily more productive on my own at home or at a cafe than I am in my office for a multitude of reasons. My purpose is to explore and explain why some people, particular management, believe remote to be less efficient.